{"thread":{"id":"49086","subject":"[RFC PATCH 0/5] filter: support for excluding all trees and blobs","startedAt":"2018-08-09T22:45:32Z","lastAt":"2018-10-16T15:00:37Z","messageCount":151,"participants":["Matthew DeVore","Jonathan Tan","Jeff Hostetler","Jeff King","Junio C Hamano","Stefan Beller","Duy Nguyen"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"355059","messageId":"cover.1533854545.git.matvore@google.com","threadId":"49086","inReplyTo":null,"subject":"[RFC PATCH 0/5] filter: support for excluding all trees and blobs","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-09T22:44:59Z","receivedAt":"2018-08-09T22:45:32Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This patch series does two things:\n\n a. (patches 1-2) introduces \"--filter=only:commits\" which filters trees and blobs\n b. (patches 3-5) better support for promisor trees in the rev-list command\n\nThe intention is to enable initial partial clones to be very tiny by only\nincluding commits. Patches 3-5 are necessary because, even though it has already\nbeen possible to have partial clones with trees missing (there are tests for it),\nthere have not been any filters which support this yet, so it seemed necessary\nto make rev-list handle this case better.\n\nThank you,\n\nMatthew DeVore (5):\n revision: invert meaning of the USER_GIVEN flag\n list-objects-filter: implement filter only:commits\n list-objects: store common func args in struct\n list-objects: refactor to process_tree_contents\n rev-list: handle missing tree objects properly\n\n Documentation/rev-list-options.txt | 2 +\n builtin/rev-list.c | 12 +-\n list-objects-filter-options.c | 4 +\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 43 +++--\n list-objects.c | 226 +++++++++++++------------\n revision.c | 1 -\n revision.h | 11 +-\n t/t5317-pack-objects-filter-objects.sh | 30 ++++\n t/t5616-partial-clone.sh | 27 +++\n t/t6112-rev-list-filters-objects.sh | 13 ++\n 11 files changed, 242 insertions(+), 128 deletions(-)\n\n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355060","messageId":"87799036f82793b691e899bda4fb882a4b231c3a.1533854545.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"[PATCH 1/5] revision: invert meaning of the USER_GIVEN flag","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-09T22:45:00Z","receivedAt":"2018-08-09T22:45:36Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Abandon the previous approach of mutating all new objects implicitly in\nadd_pending_object by inverting the meaning of the bit (it is now\nNOT_USER_GIVEN) and only setting the flag when we need to.\n\nThis more accurately tracks if a tree was provided directly by the user.\nWithout this patch, the root tree of all commits were erroneously\nconsidered to be USER_GIVEN, which meant they cannot be filtered. This\ndistinction is important in the next patch.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 31 ++++++++++++++++++-------------\n revision.c | 1 -\n revision.h | 10 +++++++---\n 3 files changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex c99c47ac1..482044bda 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -48,7 +48,7 @@ static void process_blob(struct rev_info *revs,\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && filter_fn)\n \t\tr = filter_fn(LOFS_BLOB, obj,\n \t\t\t path->buf, &path->buf[pathlen],\n \t\t\t filter_data);\n@@ -133,7 +133,7 @@ static void process_tree(struct rev_info *revs,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && filter_fn)\n \t\tr = filter_fn(LOFS_BEGIN_TREE, obj,\n \t\t\t base->buf, &base->buf[baselen],\n \t\t\t filter_data);\n@@ -156,23 +156,25 @@ static void process_tree(struct rev_info *revs,\n \t\t\t\tcontinue;\n \t\t}\n \n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(revs,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n+\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\tstruct tree *t = lookup_tree(the_repository, entry.oid);\n+\t\t\tt->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_tree(revs, t, show, base, entry.path,\n \t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(revs, entry.oid->hash,\n \t\t\t\t\tshow, base, entry.path,\n \t\t\t\t\tcb_data);\n-\t\telse\n-\t\t\tprocess_blob(revs,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n+\t\telse {\n+\t\t\tstruct blob *b = lookup_blob(the_repository, entry.oid);\n+\t\t\tb->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_blob(revs, b, show, base, entry.path,\n \t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t}\n \t}\n \n-\tif (!(obj->flags & USER_GIVEN) && filter_fn) {\n+\tif ((obj->flags & NOT_USER_GIVEN) && filter_fn) {\n \t\tr = filter_fn(LOFS_END_TREE, obj,\n \t\t\t base->buf, &base->buf[baselen],\n \t\t\t filter_data);\n@@ -301,8 +303,11 @@ static void do_traverse(struct rev_info *revs,\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n-\t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(revs, get_commit_tree(commit));\n+\t\tif (get_commit_tree(commit)) {\n+\t\t\tstruct tree *tree = get_commit_tree(commit);\n+\t\t\ttree->object.flags |= NOT_USER_GIVEN;\n+\t\t\tadd_pending_tree(revs, tree);\n+\t\t}\n \t\tshow_commit(commit, show_data);\n \n \t\tif (revs->tree_blobs_in_commit_order)\ndiff --git a/revision.c b/revision.c\nindex 062749437..6d355b43c 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -175,7 +175,6 @@ static void add_pending_object_with_path(struct rev_info *revs,\n \t\tstrbuf_release(&buf);\n \t\treturn; /* do not add the commit itself */\n \t}\n-\tobj->flags |= USER_GIVEN;\n \tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n }\n \ndiff --git a/revision.h b/revision.h\nindex c599c34da..cd6b62313 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -8,7 +8,11 @@\n #include \"diff.h\"\n #include \"commit-slab-decl.h\"\n \n-/* Remember to update object flag allocation in object.h */\n+/* Remember to update object flag allocation in object.h\n+ * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n+ * filtering trees and blobs, but it may be useful to support filtering commits\n+ * in the future.\n+ */\n #define SEEN\t\t(1u<<0)\n #define UNINTERESTING (1u<<1)\n #define TREESAME\t(1u<<2)\n@@ -20,9 +24,9 @@\n #define SYMMETRIC_LEFT\t(1u<<8)\n #define PATCHSAME\t(1u<<9)\n #define BOTTOM\t\t(1u<<10)\n-#define USER_GIVEN\t(1u<<25) /* given directly by the user */\n+#define NOT_USER_GIVEN\t(1u<<25) /* tree or blob not given directly by user */\n #define TRACK_LINEAR\t(1u<<26)\n-#define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n+#define ALL_REV_FLAGS\t(((1u<<11)-1) | NOT_USER_GIVEN | TRACK_LINEAR)\n \n #define DECORATE_SHORT_REFS\t1\n #define DECORATE_FULL_REFS\t2\n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355061","messageId":"71619cbd1004d7acf991011fc0c941d4a5716079.1533854545.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"[PATCH 2/5] list-objects-filter: implement filter only:commits","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-09T22:45:01Z","receivedAt":"2018-08-09T22:45:40Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Teach list-objects the \"only:commits\" filter which allows for filtering\nout all non-commit and non-annotated tag objects (unless other objects\nare explicitly specified by the user). The purpose of this patch is to\nallow smaller partial clones.\n\nThe name of this filter - only:commits - is a bit inaccurate because it\nstill allows annotated tags to pass through. I chose it because it was\nthe only concise name I could think of that was pretty descriptive. I\nconsidered and decided against \"tree:none\" because the code and\ndocumentation for filters seems to lack the concept of \"you're filtering\nthis, so we'll implicitly filter all referents of this.\" So \"tree:none\"\nis vague, since some may think it filters blobs too, while some may not.\n\"only:commits\" is specific and makes it easier to match it to a\npotential use case.\n\nSigned-off-by: Matthew DeVore \n---\n Documentation/rev-list-options.txt | 2 ++\n list-objects-filter-options.c | 4 +++\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 43 ++++++++++++++++++--------\n t/t5317-pack-objects-filter-objects.sh | 30 ++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 13 ++++++++\n 6 files changed, 80 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 7b273635d..3a60a490a 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -743,6 +743,8 @@ specification contained in .\n \tA debug option to help with future \"partial clone\" development.\n \tThis option specifies how missing objects are handled.\n +\n+The form '--filter=only:commits' omits all blobs and trees.\n++\n The form '--missing=error' requests that rev-list stop with an error if\n a missing object is encountered. This is the default action.\n +\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex c0e2bd6a0..aaaaae508 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -69,6 +69,10 @@ static int gently_parse_list_objects_filter(\n \t\tfilter_options->choice = LOFC_SPARSE_PATH;\n \t\tfilter_options->sparse_path_value = strdup(v0);\n \t\treturn 0;\n+\n+\t} else if (!strcmp(arg, \"only:commits\")) {\n+\t\tfilter_options->choice = LOFC_ONLY_COMMITS;\n+\t\treturn 0;\n \t}\n \n \tif (errbuf) {\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 0000a61f8..a68df42c8 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -12,6 +12,7 @@ enum list_objects_filter_choice {\n \tLOFC_BLOB_LIMIT,\n \tLOFC_SPARSE_OID,\n \tLOFC_SPARSE_PATH,\n+\tLOFC_ONLY_COMMITS,\n \tLOFC__COUNT /* must be last */\n };\n \ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex a0ba78b20..f0a064b4b 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -26,38 +26,39 @@\n #define FILTER_SHOWN_BUT_REVISIT (1<<21)\n \n /*\n- * A filter for list-objects to omit ALL blobs from the traversal.\n- * And to OPTIONALLY collect a list of the omitted OIDs.\n+ * A filter for list-objects to omit ALL blobs from the traversal, and possibly\n+ * trees as well.\n+ * Can OPTIONALLY collect a list of the omitted OIDs.\n */\n-struct filter_blobs_none_data {\n+struct filter_none_of_type_data {\n+\tunsigned omit_trees : 1;\n \tstruct oidset *omits;\n };\n \n-static enum list_objects_filter_result filter_blobs_none(\n+static enum list_objects_filter_result filter_none_of_type(\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n \tvoid *filter_data_)\n {\n-\tstruct filter_blobs_none_data *filter_data = filter_data_;\n+\tstruct filter_none_of_type_data *filter_data = filter_data_;\n \n \tswitch (filter_situation) {\n \tdefault:\n \t\tdie(\"unknown filter_situation\");\n \t\treturn LOFR_ZERO;\n \n-\tcase LOFS_BEGIN_TREE:\n-\t\tassert(obj->type == OBJ_TREE);\n-\t\t/* always include all tree objects */\n-\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n-\n \tcase LOFS_END_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\treturn LOFR_ZERO;\n \n+\tcase LOFS_BEGIN_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\tif (!filter_data->omit_trees)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_BLOB:\n-\t\tassert(obj->type == OBJ_BLOB);\n \t\tassert((obj->flags & SEEN) == 0);\n \n \t\tif (filter_data->omits)\n@@ -72,10 +73,25 @@ static void *filter_blobs_none__init(\n \tfilter_object_fn *filter_fn,\n \tfilter_free_fn *filter_free_fn)\n {\n-\tstruct filter_blobs_none_data *d = xcalloc(1, sizeof(*d));\n+\tstruct filter_none_of_type_data *d = xcalloc(1, sizeof(*d));\n+\td->omits = omitted;\n+\n+\t*filter_fn = filter_none_of_type;\n+\t*filter_free_fn = free;\n+\treturn d;\n+}\n+\n+static void* filter_only_commits__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn)\n+{\n+\tstruct filter_none_of_type_data *d = xcalloc(1, sizeof(*d));\n+\td->omit_trees = 1;\n \td->omits = omitted;\n \n-\t*filter_fn = filter_blobs_none;\n+\t*filter_fn = filter_none_of_type;\n \t*filter_free_fn = free;\n \treturn d;\n }\n@@ -376,6 +392,7 @@ static filter_init_fn s_filters[] = {\n \tfilter_blobs_limit__init,\n \tfilter_sparse_oid__init,\n \tfilter_sparse_path__init,\n+\tfilter_only_commits__init,\n };\n \n void *list_objects_filter__init(\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 6710c8bc8..600d153f9 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -59,6 +59,36 @@ test_expect_success 'verify normal and blob:none packfiles have same commits/tre\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'setup for tests of only:commits' '\n+\tmkdir r1/subtree &&\n+\techo \"This is a file in a subtree\" > r1/subtree/file &&\n+\tgit -C r1 add subtree/file &&\n+\tgit -C r1 commit -m subtree\n+'\n+\n+test_expect_success 'verify only:commits packfile has no blobs or trees' '\n+\tgit -C r1 pack-objects --rev --stdout --filter=only:commits >commitsonly.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack \\\n+\t\t| grep -E \"tree|blob\" \\\n+\t\t| sort >observed &&\n+\ttest_line_count = 0 observed\n+'\n+\n+test_expect_success 'grab tree directly when using only:commits' '\n+\t# We should get the tree specified directly but not its blobs or subtrees.\n+\tgit -C r1 pack-objects --rev --stdout --filter=only:commits >commitsonly.pack <<-EOF &&\n+\tHEAD:\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack \\\n+\t\t| grep -E \"tree|blob\" \\\n+\t\t| sort >observed &&\n+\ttest_line_count = 1 observed\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 0a37dd5f9..6dbd9477c 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -196,6 +196,19 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n \ttest_cmp observed expected\n '\n \n+# Test only:commits filter.\n+\n+test_expect_success 'verify only:commits includes trees in \"filtered\" output' '\n+\tgit -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=only:commits \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed s/~// \\\n+\t\t| xargs -n1 git -C r3 cat-file -t \\\n+\t\t| sort -u >filtered_types &&\n+\tprintf \"blob\\ntree\\n\" > expected &&\n+\ttest_cmp filtered_types expected\n+'\n+\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355062","messageId":"d1bdc10e2700f6d5b400ab1c06cacd50a1345f03.1533854545.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"[PATCH 3/5] list-objects: store common func args in struct","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-09T22:45:02Z","receivedAt":"2018-08-09T22:45:42Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will make utility functions easier to create, as done by the next\npatch.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 152 +++++++++++++++++++++++--------------------------\n 1 file changed, 71 insertions(+), 81 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 482044bda..fa34fbf58 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -12,20 +12,25 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n \n-static void process_blob(struct rev_info *revs,\n+struct traversal_context {\n+\tstruct rev_info *revs;\n+\tshow_object_fn show_object;\n+\tshow_commit_fn show_commit;\n+\tvoid *show_data;\n+\tfilter_object_fn filter_fn;\n+\tvoid *filter_data;\n+};\n+\n+static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \n-\tif (!revs->blob_objects)\n+\tif (!ctx->revs->blob_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad blob object\");\n@@ -41,21 +46,21 @@ static void process_blob(struct rev_info *revs,\n \t * may cause the actual filter to report an incomplete list\n \t * of missing objects.\n \t */\n-\tif (revs->exclude_promisor_objects &&\n+\tif (ctx->revs->exclude_promisor_objects &&\n \t !has_object_file(&obj->oid) &&\n \t is_promisor_object(&obj->oid))\n \t\treturn;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif ((obj->flags & NOT_USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BLOB, obj,\n-\t\t\t path->buf, &path->buf[pathlen],\n-\t\t\t filter_data);\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n+\t\t\t\t path->buf, &path->buf[pathlen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, path->buf, cb_data);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -81,26 +86,21 @@ static void process_blob(struct rev_info *revs,\n * the link, and how to do it. Whether it necessarily makes\n * any sense what-so-ever to ever do that is another issue.\n */\n-static void process_gitlink(struct rev_info *revs,\n+static void process_gitlink(struct traversal_context *ctx,\n \t\t\t const unsigned char *sha1,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data)\n+\t\t\t const char *name)\n {\n \t/* Nothing to do */\n }\n \n-static void process_tree(struct rev_info *revs,\n+static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &tree->object;\n+\tstruct rev_info *revs = ctx->revs;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n@@ -133,14 +133,14 @@ static void process_tree(struct rev_info *revs,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif ((obj->flags & NOT_USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BEGIN_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, base->buf, cb_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -159,29 +159,26 @@ static void process_tree(struct rev_info *revs,\n \t\tif (S_ISDIR(entry.mode)) {\n \t\t\tstruct tree *t = lookup_tree(the_repository, entry.oid);\n \t\t\tt->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_tree(revs, t, show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\tprocess_tree(ctx, t, base, entry.path);\n \t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(revs, entry.oid->hash,\n-\t\t\t\t\tshow, base, entry.path,\n-\t\t\t\t\tcb_data);\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash,\n+\t\t\t\t\tbase, entry.path);\n \t\telse {\n \t\t\tstruct blob *b = lookup_blob(the_repository, entry.oid);\n \t\t\tb->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_blob(revs, b, show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\tprocess_blob(ctx, b, base, entry.path);\n \t\t}\n \t}\n \n-\tif ((obj->flags & NOT_USER_GIVEN) && filter_fn) {\n-\t\tr = filter_fn(LOFS_END_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n+\t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \t\tif (r & LOFR_MARK_SEEN)\n \t\t\tobj->flags |= SEEN;\n \t\tif (r & LOFR_DO_SHOW)\n-\t\t\tshow(obj, base->buf, cb_data);\n+\t\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \t}\n \n \tstrbuf_setlen(base, baselen);\n@@ -244,19 +241,15 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n-static void traverse_trees_and_blobs(struct rev_info *revs,\n-\t\t\t\t struct strbuf *base,\n-\t\t\t\t show_object_fn show_object,\n-\t\t\t\t void *show_data,\n-\t\t\t\t filter_object_fn filter_fn,\n-\t\t\t\t void *filter_data)\n+static void traverse_trees_and_blobs(struct traversal_context *ctx,\n+\t\t\t\t struct strbuf *base)\n {\n \tint i;\n \n \tassert(base->len == 0);\n \n-\tfor (i = 0; i < revs->pending.nr; i++) {\n-\t\tstruct object_array_entry *pending = revs->pending.objects + i;\n+\tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n+\t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n@@ -264,41 +257,32 @@ static void traverse_trees_and_blobs(struct rev_info *revs,\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tshow_object(obj, name, show_data);\n+\t\t\tctx->show_object(obj, name, ctx->show_data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(revs, (struct tree *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(revs, (struct blob *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n \t\t oid_to_hex(&obj->oid), name);\n \t}\n-\tobject_array_clear(&revs->pending);\n+\tobject_array_clear(&ctx->revs->pending);\n }\n \n-static void do_traverse(struct rev_info *revs,\n-\t\t\tshow_commit_fn show_commit,\n-\t\t\tshow_object_fn show_object,\n-\t\t\tvoid *show_data,\n-\t\t\tfilter_object_fn filter_fn,\n-\t\t\tvoid *filter_data)\n+static void do_traverse(struct traversal_context *ctx)\n {\n \tstruct commit *commit;\n \tstruct strbuf csp; /* callee's scratch pad */\n \tstrbuf_init(&csp, PATH_MAX);\n \n-\twhile ((commit = get_revision(revs)) != NULL) {\n+\twhile ((commit = get_revision(ctx->revs)) != NULL) {\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n@@ -306,23 +290,19 @@ static void do_traverse(struct rev_info *revs,\n \t\tif (get_commit_tree(commit)) {\n \t\t\tstruct tree *tree = get_commit_tree(commit);\n \t\t\ttree->object.flags |= NOT_USER_GIVEN;\n-\t\t\tadd_pending_tree(revs, tree);\n+\t\t\tadd_pending_tree(ctx->revs, tree);\n \t\t}\n-\t\tshow_commit(commit, show_data);\n+\t\tctx->show_commit(commit, ctx->show_data);\n \n-\t\tif (revs->tree_blobs_in_commit_order)\n+\t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n \t\t\t * NEEDSWORK: Adding the tree and then flushing it here\n \t\t\t * needs a reallocation for each commit. Can we pass the\n \t\t\t * tree directory without allocation churn?\n \t\t\t */\n-\t\t\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t\t\t show_object, show_data,\n-\t\t\t\t\t\t filter_fn, filter_data);\n+\t\t\ttraverse_trees_and_blobs(ctx, &csp);\n \t}\n-\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t show_object, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\ttraverse_trees_and_blobs(ctx, &csp);\n \tstrbuf_release(&csp);\n }\n \n@@ -331,7 +311,14 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t\t show_object_fn show_object,\n \t\t\t void *show_data)\n {\n-\tdo_traverse(revs, show_commit, show_object, show_data, NULL, NULL);\n+\tstruct traversal_context ctx;\n+\tctx.revs = revs;\n+\tctx.show_commit = show_commit;\n+\tctx.show_object = show_object;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\tctx.filter_data = NULL;\n+\tdo_traverse(&ctx);\n }\n \n void traverse_commit_list_filtered(\n@@ -342,14 +329,17 @@ void traverse_commit_list_filtered(\n \tvoid *show_data,\n \tstruct oidset *omitted)\n {\n-\tfilter_object_fn filter_fn = NULL;\n+\tstruct traversal_context ctx;\n+\tctx.revs = revs;\n+\tctx.show_object = show_object;\n+\tctx.show_commit = show_commit;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n \tfilter_free_fn filter_free_fn = NULL;\n-\tvoid *filter_data = NULL;\n \n-\tfilter_data = list_objects_filter__init(omitted, filter_options,\n-\t\t\t\t\t\t&filter_fn, &filter_free_fn);\n-\tdo_traverse(revs, show_commit, show_object, show_data,\n-\t\t filter_fn, filter_data);\n-\tif (filter_data && filter_free_fn)\n-\t\tfilter_free_fn(filter_data);\n+\tctx.filter_data = list_objects_filter__init(omitted, filter_options,\n+\t\t\t\t\t\t &ctx.filter_fn, &filter_free_fn);\n+\tdo_traverse(&ctx);\n+\tif (ctx.filter_data && filter_free_fn)\n+\t\tfilter_free_fn(ctx.filter_data);\n }\n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355063","messageId":"b3d4e955b66aa1e871ad4bfab002ca5ebbd3d972.1533854545.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"[PATCH 4/5] list-objects: refactor to process_tree_contents","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-09T22:45:03Z","receivedAt":"2018-08-09T22:45:45Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will be used in a follow-up patch to reduce indentation needed when\ninvoking the logic conditionally. i.e. rather than:\n\nif (foo) {\n\twhile (...) {\n\t\t/* this is very indented */\n\t}\n}\n\nwe will have:\n\nif (foo)\n\tprocess_tree_contents(...);\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 73 +++++++++++++++++++++++++++++---------------------\n 1 file changed, 43 insertions(+), 30 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex fa34fbf58..7ecdb95ce 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -94,6 +94,48 @@ static void process_gitlink(struct traversal_context *ctx,\n \t/* Nothing to do */\n }\n \n+static void process_tree(struct traversal_context *ctx,\n+\t\t\t struct tree *tree,\n+\t\t\t struct strbuf *base,\n+\t\t\t const char *name);\n+\n+static void process_tree_contents(struct traversal_context *ctx,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *base)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tenum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ?\n+\t\tall_entries_interesting : entry_not_interesting;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tif (match != all_entries_interesting) {\n+\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n+\t\t\t\t\t\t &ctx->revs->diffopt.pathspec);\n+\t\t\tif (match == all_entries_not_interesting)\n+\t\t\t\tbreak;\n+\t\t\tif (match == entry_not_interesting)\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\tstruct tree *t = lookup_tree(the_repository, entry.oid);\n+\t\t\tt->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_tree(ctx, t, base, entry.path);\n+\t\t}\n+\t\telse if (S_ISGITLINK(entry.mode))\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash,\n+\t\t\t\t\tbase, entry.path);\n+\t\telse {\n+\t\t\tstruct blob *b = lookup_blob(the_repository, entry.oid);\n+\t\t\tb->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_blob(ctx, b, base, entry.path);\n+\t\t}\n+\t}\n+}\n+\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n@@ -101,10 +143,6 @@ static void process_tree(struct traversal_context *ctx,\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n-\tstruct tree_desc desc;\n-\tstruct name_entry entry;\n-\tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n-\t\tall_entries_interesting: entry_not_interesting;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \tint gently = revs->ignore_missing_links ||\n@@ -144,32 +182,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tinit_tree_desc(&desc, tree->buffer, tree->size);\n-\n-\twhile (tree_entry(&desc, &entry)) {\n-\t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n-\t\t\t\t\t\t &revs->diffopt.pathspec);\n-\t\t\tif (match == all_entries_not_interesting)\n-\t\t\t\tbreak;\n-\t\t\tif (match == entry_not_interesting)\n-\t\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (S_ISDIR(entry.mode)) {\n-\t\t\tstruct tree *t = lookup_tree(the_repository, entry.oid);\n-\t\t\tt->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_tree(ctx, t, base, entry.path);\n-\t\t}\n-\t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(ctx, entry.oid->hash,\n-\t\t\t\t\tbase, entry.path);\n-\t\telse {\n-\t\t\tstruct blob *b = lookup_blob(the_repository, entry.oid);\n-\t\t\tb->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_blob(ctx, b, base, entry.path);\n-\t\t}\n-\t}\n+\tprocess_tree_contents(ctx, tree, base);\n \n \tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355064","messageId":"54d827e7c0dc757451fa10f5bd0752e1e3356281.1533854545.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"[PATCH 5/5] rev-list: handle missing tree objects properly","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-09T22:45:04Z","receivedAt":"2018-08-09T22:45:47Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Previously, we assumed only blob objects could be missing. This patch\nmakes rev-list handle missing trees like missing blobs. A missing tree\nwill cause an error if --missing indicates an error should be caused,\nand the hash is printed even if the tree is missing.\n\nSigned-off-by: Matthew DeVore \n---\n builtin/rev-list.c | 12 ++++++++----\n list-objects.c | 8 ++++++--\n revision.h | 1 +\n t/t5616-partial-clone.sh | 27 +++++++++++++++++++++++++++\n 4 files changed, 42 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 5b07f3f4a..c870d4fe6 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -6,6 +6,7 @@\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"object.h\"\n #include \"object-store.h\"\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n@@ -209,7 +210,8 @@ static inline void finish_object__ma(struct object *obj)\n \t */\n \tswitch (arg_missing_action) {\n \tcase MA_ERROR:\n-\t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n+\t\tdie(\"missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tcase MA_ALLOW_ANY:\n@@ -222,8 +224,8 @@ static inline void finish_object__ma(struct object *obj)\n \tcase MA_ALLOW_PROMISOR:\n \t\tif (is_promisor_object(&obj->oid))\n \t\t\treturn;\n-\t\tdie(\"unexpected missing blob object '%s'\",\n-\t\t oid_to_hex(&obj->oid));\n+\t\tdie(\"unexpected missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tdefault:\n@@ -235,7 +237,7 @@ static inline void finish_object__ma(struct object *obj)\n static int finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\tif (!has_object_file(&obj->oid)) {\n \t\tfinish_object__ma(obj);\n \t\treturn 1;\n \t}\n@@ -373,6 +375,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tinit_revisions(&revs, prefix);\n \trevs.abbrev = DEFAULT_ABBREV;\n \trevs.commit_format = CMIT_FMT_UNSPECIFIED;\n+\trevs.show_missing_trees = 1;\n \n \t/*\n \t * Scan the argument list before invoking setup_revisions(), so that we\n@@ -389,6 +392,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\tif (!strcmp(arg, \"--exclude-promisor-objects\")) {\n \t\t\tfetch_if_missing = 0;\n \t\t\trevs.exclude_promisor_objects = 1;\n+\t\t\trevs.show_missing_trees = 0;\n \t\t\tbreak;\n \t\t}\n \t}\ndiff --git a/list-objects.c b/list-objects.c\nindex 7ecdb95ce..b0291c45a 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -146,7 +146,9 @@ static void process_tree(struct traversal_context *ctx,\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \tint gently = revs->ignore_missing_links ||\n+\t\t revs->show_missing_trees ||\n \t\t revs->exclude_promisor_objects;\n+\tint parse_result;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -154,7 +156,8 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, gently) < 0) {\n+\tparse_result = parse_tree_gently(tree, gently);\n+\tif (parse_result < 0 && !revs->show_missing_trees) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n@@ -182,7 +185,8 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tprocess_tree_contents(ctx, tree, base);\n+\tif (parse_result >= 0)\n+\t\tprocess_tree_contents(ctx, tree, base);\n \n \tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\ndiff --git a/revision.h b/revision.h\nindex cd6b62313..34ff99f05 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -128,6 +128,7 @@ struct rev_info {\n \t\t\tfirst_parent_only:1,\n \t\t\tline_level_traverse:1,\n \t\t\ttree_blobs_in_commit_order:1,\n+\t\t\tshow_missing_trees:1,\n \n \t\t\t/* for internal use only */\n \t\t\texclude_promisor_objects:1;\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex bbbe7537d..8a0ca0a74 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -170,6 +170,33 @@ test_expect_success 'partial clone fetches blobs pointed to by refs even if norm\n \tgit -C dst fsck\n '\n \n+test_expect_success 'can use only:commits to filter partial clone' '\n+\trm -rf dst &&\n+\tgit clone --no-checkout --filter=only:commits \"file://$(pwd)/srv.bare\" dst &&\n+\tgit -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n+\tcat fetched_objects \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n+\tsort fetched_types -u >unique_types.observed &&\n+\techo commit > unique_types.expected &&\n+\ttest_cmp unique_types.observed unique_types.expected\n+'\n+\n+test_expect_success 'show missing tree objects with --missing=print' '\n+\tgit -C dst rev-list master --missing=print --quiet --objects >missing_objs &&\n+\tsed \"s/?//\" missing_objs \\\n+\t\t| xargs -n1 git -C srv.bare cat-file -t \\\n+\t\t>missing_types &&\n+\tsort -u missing_types >missing_types.uniq &&\n+\techo tree >expected &&\n+\ttest_cmp missing_types.uniq expected\n+'\n+\n+test_expect_success 'do not complain when a missing tree cannot be parsed' '\n+\tgit -C dst rev-list master --missing=print --quiet --objects 2>rev_list_err >&2 &&\n+\t! grep -q \"Could not read \" rev_list_err\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355071","messageId":"20180810001447.11257-1-jonathantanmy@google.com","threadId":"49086","inReplyTo":"71619cbd1004d7acf991011fc0c941d4a5716079.1533854545.git.matvore@google.com","subject":"Re: [PATCH 2/5] list-objects-filter: implement filter only:commits","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-08-10T00:14:47Z","receivedAt":"2018-08-10T00:14:54Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Teach list-objects the \"only:commits\" filter which allows for filtering\n> out all non-commit and non-annotated tag objects (unless other objects\n> are explicitly specified by the user). The purpose of this patch is to\n> allow smaller partial clones.\n> \n> The name of this filter - only:commits - is a bit inaccurate because it\n> still allows annotated tags to pass through. I chose it because it was\n> the only concise name I could think of that was pretty descriptive. I\n> considered and decided against \"tree:none\" because the code and\n> documentation for filters seems to lack the concept of \"you're filtering\n> this, so we'll implicitly filter all referents of this.\" So \"tree:none\"\n> is vague, since some may think it filters blobs too, while some may not.\n> \"only:commits\" is specific and makes it easier to match it to a\n> potential use case.\n\nI'll do a fuller review tomorrow, but here are my initial thoughts.\n\nI'm undecided about whether \"only:commits\" or \"tree:none\" is better -\none argument in favor of the latter is that blobs are not of much use\nwithout any trees referring to them, so it makes sense that omitting\ntrees means omitting blobs. But that requires some thought and is not\nimmediately obvious.\n\n> /*\n> - * A filter for list-objects to omit ALL blobs from the traversal.\n> - * And to OPTIONALLY collect a list of the omitted OIDs.\n> + * A filter for list-objects to omit ALL blobs from the traversal, and possibly\n> + * trees as well.\n> + * Can OPTIONALLY collect a list of the omitted OIDs.\n> */\n> -struct filter_blobs_none_data {\n> +struct filter_none_of_type_data {\n> +\tunsigned omit_trees : 1;\n> \tstruct oidset *omits;\n> };\n\nI know that it's documented above that blobs are always omitted, but\nmaybe it's worth it to add a comment /* blobs are always omitted */.\n\n> -\tcase LOFS_BEGIN_TREE:\n> -\t\tassert(obj->type == OBJ_TREE);\n> -\t\t/* always include all tree objects */\n> -\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> -\n> \tcase LOFS_END_TREE:\n> \t\tassert(obj->type == OBJ_TREE);\n> \t\treturn LOFR_ZERO;\n> \n> +\tcase LOFS_BEGIN_TREE:\n> +\t\tassert(obj->type == OBJ_TREE);\n> +\t\tif (!filter_data->omit_trees)\n> +\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> +\n> \tcase LOFS_BLOB:\n> -\t\tassert(obj->type == OBJ_BLOB);\n> \t\tassert((obj->flags & SEEN) == 0);\n\nMoving the case LOFS_BEGIN_TREE and removing the assert is unnecessary,\nI think.\n\nAlso, there's fallthrough. If that's on purpose, add /* fallthrough */,\nalthough I think that it complicates the code unnecessarily here.\n\n> +test_expect_success 'verify only:commits packfile has no blobs or trees' '\n> +\tgit -C r1 pack-objects --rev --stdout --filter=only:commits >commitsonly.pack <<-EOF &&\n> +\tHEAD\n> +\tEOF\n> +\tgit -C r1 index-pack ../commitsonly.pack &&\n> +\tgit -C r1 verify-pack -v ../commitsonly.pack \\\n> +\t\t| grep -E \"tree|blob\" \\\n> +\t\t| sort >observed &&\n> +\ttest_line_count = 0 observed\n> +'\n\nBash pipes conceal return codes. Here it's OK, but it might be better to\nwrite the verify-pack on its own line and then '! grep -E \"tree|blob\"' -\nyou don't need to sort or test_line_count.\n\n> +test_expect_success 'grab tree directly when using only:commits' '\n> +\t# We should get the tree specified directly but not its blobs or subtrees.\n> +\tgit -C r1 pack-objects --rev --stdout --filter=only:commits >commitsonly.pack <<-EOF &&\n> +\tHEAD:\n> +\tEOF\n> +\tgit -C r1 index-pack ../commitsonly.pack &&\n> +\tgit -C r1 verify-pack -v ../commitsonly.pack \\\n> +\t\t| grep -E \"tree|blob\" \\\n> +\t\t| sort >observed &&\n> +\ttest_line_count = 1 observed\n> +'\n\nSimilar comment as above, except you can redirect the output of grep to\na file, then test_line_count on that file. No need for sort.\n"},{"id":"355072","messageId":"20180810002411.13447-1-jonathantanmy@google.com","threadId":"49086","inReplyTo":"54d827e7c0dc757451fa10f5bd0752e1e3356281.1533854545.git.matvore@google.com","subject":"Re: [PATCH 5/5] rev-list: handle missing tree objects properly","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-08-10T00:24:11Z","receivedAt":"2018-08-10T00:24:16Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> @@ -209,7 +210,8 @@ static inline void finish_object__ma(struct object *obj)\n> \t */\n> \tswitch (arg_missing_action) {\n> \tcase MA_ERROR:\n> -\t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n> +\t\tdie(\"missing %s object '%s'\",\n> +\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n> \t\treturn;\n> \n> \tcase MA_ALLOW_ANY:\n> @@ -222,8 +224,8 @@ static inline void finish_object__ma(struct object *obj)\n> \tcase MA_ALLOW_PROMISOR:\n> \t\tif (is_promisor_object(&obj->oid))\n> \t\t\treturn;\n> -\t\tdie(\"unexpected missing blob object '%s'\",\n> -\t\t oid_to_hex(&obj->oid));\n> +\t\tdie(\"unexpected missing %s object '%s'\",\n> +\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n> \t\treturn;\n\nOnce again, I'll do a fuller review tomorrow.\n\nThese are fine (obj->type is populated), because the types of objects\nare known during traversal.\n\n> -\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n> +\tif (!has_object_file(&obj->oid)) {\n> \t\tfinish_object__ma(obj);\n> \t\treturn 1;\n\nAnd this is also fine, because finish_object__ma can now handle any\nobject type.\n\n> +\trevs.show_missing_trees = 1;\n\n(and elsewhere)\n\nCould we just show missing trees all the time? We do that for blobs and\nalready rely on the caller (eventually, show_object() in\nbuiltin/rev-list.c) to determine whether the object actually exists or\nnot; we could do the same for trees. This allows us to not include this\nextra knob.\n\n> -\tif (parse_tree_gently(tree, gently) < 0) {\n> +\tparse_result = parse_tree_gently(tree, gently);\n> +\tif (parse_result < 0 && !revs->show_missing_trees) {\n> \t\tif (revs->ignore_missing_links)\n> \t\t\treturn;\n> \n> @@ -182,7 +185,8 @@ static void process_tree(struct traversal_context *ctx,\n> \tif (base->len)\n> \t\tstrbuf_addch(base, '/');\n> \n> -\tprocess_tree_contents(ctx, tree, base);\n> +\tif (parse_result >= 0)\n> +\t\tprocess_tree_contents(ctx, tree, base);\n> \n> \tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n> \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n\nIs it possible to call the appropriate callbacks and then return\nimmediately, instead of going through the whole function checking\nparse_result when necessary? When doing the latter, the reader needs to\nkeep on checking if each function still works if the tree is\nunparseable.\n"},{"id":"355130","messageId":"20180810184341.208183-1-jonathantanmy@google.com","threadId":"49086","inReplyTo":"87799036f82793b691e899bda4fb882a4b231c3a.1533854545.git.matvore@google.com","subject":"Re: [PATCH 1/5] revision: invert meaning of the USER_GIVEN flag","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-08-10T18:43:41Z","receivedAt":"2018-08-10T18:43:48Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Abandon the previous approach of mutating all new objects implicitly in\n> add_pending_object by inverting the meaning of the bit (it is now\n> NOT_USER_GIVEN) and only setting the flag when we need to.\n> \n> This more accurately tracks if a tree was provided directly by the user.\n> Without this patch, the root tree of all commits were erroneously\n> considered to be USER_GIVEN, which meant they cannot be filtered. This\n> distinction is important in the next patch.\n\nAfter rereading this patch, I think the thought process is:\n\n - the existing code inaccurately makes root trees of commits USER_GIVEN\n - instead of trying to fix that, it is easier to invert the meaning of this\n flag, and since we only need to track trees and blobs, let's do so in this\n patch\n\nSo a better commit message might be:\n\n revision: mark non-user-given objects instead\n\n Currently, list-objects.c incorrectly treats all root trees of commits\n as USER_GIVEN. Also, it would be easier to mark objects that are\n non-user-given instead of user-given, since the places in the code\n where we access an object through a reference are more obvious than\n the places where we access an object that was given by the user.\n\n Resolve these two problems by introducing a flag NOT_USER_GIVEN that\n marks blobs and trees that are non-user-given, replacing USER_GIVEN.\n (Only blobs and trees are marked because this mark is only used when\n filtering objects, and filtering of other types of objects are not\n supported yet.)\n\nThe patch itself looks good to me.\n"},{"id":"355133","messageId":"20180810190352.62705-1-jonathantanmy@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"Re: [RFC PATCH 0/5] filter: support for excluding all trees and blobs","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-08-10T19:03:52Z","receivedAt":"2018-08-10T19:04:00Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Matthew DeVore (5):\n> revision: invert meaning of the USER_GIVEN flag\n> list-objects-filter: implement filter only:commits\n> list-objects: store common func args in struct\n> list-objects: refactor to process_tree_contents\n> rev-list: handle missing tree objects properly\n\nFirstly, run every patch with \"make DEVELOPER=1\" - there is at least one\n\"mixed declarations and code\", which the Git coding style does not\nallow.\n\nI've already replied to patches 1, 2, and 5. Patches 3 and 4 look OK to\nme and seem like good changes (patch 4, in addition to reducing\nindentation, also reduces the scope of the local variables - so it is a\ngood change).\n\nOne last thing is that I'm not sure that this order of patches is the\nbest order - in particular, if I run the tests at the 5th patch using a\nbinary compiled at the 4th patch, I notice that cloning with\n\"--filter=only:commits\" fails with a cryptic error \"fatal: bad tree\nobject e891efadd67ca0c01b1c518a2fd91130d40f5904\". This makes bisecting\nfor errors difficult, but perhaps with this problem manifesting in only\na few commits, it is not so bad.\n\nThe ideal order is to put patches 3-5 before 1-2. I've tried the\nrearrangement myself and found many instances where I had to rewrite\ncode because one patch introduces \"ctx\" and the other, NOT_USER_GIVEN.\nSo as a reviewer, I'm on the fence about suggesting that the patches be\nreordered.\n"},{"id":"355209","messageId":"cover.1533941986.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"[PATCH v2 0/5] filter: support for excluding all trees and blobs","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-10T23:06:24Z","receivedAt":"2018-08-10T23:06:50Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Changes applied, as suggested by jonathantanmy@google.com:\n- Re-ordered patches so 3-5 actually come first\n- Sadly, as a result of the above, many of the tests in the \"treat\n missing trees like missing blobs\" patch had to be moved to the filter\n implementation patch, since it doesn't seem possible to create\n promisor objects that are really recognized as promisor objects in\n tests (unless you actually do a partial clone). Overall, I thought\n this ordering was more elegant, so I kept it.\n- Reworded NOT_USER_GIVEN commit message as suggested\n- Fixed style error list-objects.c (var dec and code mixed)\n- Added missing /* fallthrough */ and explanation why\n- Removed the show_missing_trees flag - now we won't show any error if\n the only problem is the object is missing and it's a promisor object.\n- Renamed only:commits to tree:none and updated commit message\n accordingly\n- Added /* blobs are always omitted */ comment in list-objects-filter.c\n- Fixed up tests in t5317-pack-objects-filter-objects.sh to not use\n unnecessary sorts, and to do more commands on lines of their own\n rather than in pipes\n\n\nMatthew DeVore (5):\n list-objects: store common func args in struct\n list-objects: refactor to process_tree_contents\n rev-list: handle missing tree objects properly\n revision: mark non-user-given objects instead\n list-objects-filter: implement filter tree:none\n\n Documentation/rev-list-options.txt | 2 +\n builtin/rev-list.c | 10 +-\n list-objects-filter-options.c | 4 +\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 49 +++--\n list-objects.c | 236 +++++++++++++------------\n revision.c | 1 -\n revision.h | 10 +-\n t/t5317-pack-objects-filter-objects.sh | 40 +++++\n t/t5616-partial-clone.sh | 27 +++\n t/t6112-rev-list-filters-objects.sh | 13 ++\n 11 files changed, 259 insertions(+), 134 deletions(-)\n\n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355210","messageId":"c54a2818125739b7dde40aeee2cf1d72e057e963.1533941986.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533941986.git.matvore@google.com","subject":"[PATCH v2 1/5] list-objects: store common func args in struct","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-10T23:06:25Z","receivedAt":"2018-08-10T23:06:52Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will make utility functions easier to create, as done by the next\npatch.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 158 +++++++++++++++++++++++--------------------------\n 1 file changed, 74 insertions(+), 84 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex c99c47ac1..584518a3f 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -12,20 +12,25 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n \n-static void process_blob(struct rev_info *revs,\n+struct traversal_context {\n+\tstruct rev_info *revs;\n+\tshow_object_fn show_object;\n+\tshow_commit_fn show_commit;\n+\tvoid *show_data;\n+\tfilter_object_fn filter_fn;\n+\tvoid *filter_data;\n+};\n+\n+static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \n-\tif (!revs->blob_objects)\n+\tif (!ctx->revs->blob_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad blob object\");\n@@ -41,21 +46,21 @@ static void process_blob(struct rev_info *revs,\n \t * may cause the actual filter to report an incomplete list\n \t * of missing objects.\n \t */\n-\tif (revs->exclude_promisor_objects &&\n+\tif (ctx->revs->exclude_promisor_objects &&\n \t !has_object_file(&obj->oid) &&\n \t is_promisor_object(&obj->oid))\n \t\treturn;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BLOB, obj,\n-\t\t\t path->buf, &path->buf[pathlen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n+\t\t\t\t path->buf, &path->buf[pathlen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, path->buf, cb_data);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -81,26 +86,21 @@ static void process_blob(struct rev_info *revs,\n * the link, and how to do it. Whether it necessarily makes\n * any sense what-so-ever to ever do that is another issue.\n */\n-static void process_gitlink(struct rev_info *revs,\n+static void process_gitlink(struct traversal_context *ctx,\n \t\t\t const unsigned char *sha1,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data)\n+\t\t\t const char *name)\n {\n \t/* Nothing to do */\n }\n \n-static void process_tree(struct rev_info *revs,\n+static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &tree->object;\n+\tstruct rev_info *revs = ctx->revs;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n@@ -133,14 +133,14 @@ static void process_tree(struct rev_info *revs,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BEGIN_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, base->buf, cb_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -157,29 +157,25 @@ static void process_tree(struct rev_info *revs,\n \t\t}\n \n \t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(revs,\n+\t\t\tprocess_tree(ctx,\n \t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(revs, entry.oid->hash,\n-\t\t\t\t\tshow, base, entry.path,\n-\t\t\t\t\tcb_data);\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n \t\telse\n-\t\t\tprocess_blob(revs,\n+\t\t\tprocess_blob(ctx,\n \t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t}\n \n-\tif (!(obj->flags & USER_GIVEN) && filter_fn) {\n-\t\tr = filter_fn(LOFS_END_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \t\tif (r & LOFR_MARK_SEEN)\n \t\t\tobj->flags |= SEEN;\n \t\tif (r & LOFR_DO_SHOW)\n-\t\t\tshow(obj, base->buf, cb_data);\n+\t\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \t}\n \n \tstrbuf_setlen(base, baselen);\n@@ -242,19 +238,15 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n-static void traverse_trees_and_blobs(struct rev_info *revs,\n-\t\t\t\t struct strbuf *base,\n-\t\t\t\t show_object_fn show_object,\n-\t\t\t\t void *show_data,\n-\t\t\t\t filter_object_fn filter_fn,\n-\t\t\t\t void *filter_data)\n+static void traverse_trees_and_blobs(struct traversal_context *ctx,\n+\t\t\t\t struct strbuf *base)\n {\n \tint i;\n \n \tassert(base->len == 0);\n \n-\tfor (i = 0; i < revs->pending.nr; i++) {\n-\t\tstruct object_array_entry *pending = revs->pending.objects + i;\n+\tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n+\t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n@@ -262,62 +254,49 @@ static void traverse_trees_and_blobs(struct rev_info *revs,\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tshow_object(obj, name, show_data);\n+\t\t\tctx->show_object(obj, name, ctx->show_data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(revs, (struct tree *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(revs, (struct blob *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n \t\t oid_to_hex(&obj->oid), name);\n \t}\n-\tobject_array_clear(&revs->pending);\n+\tobject_array_clear(&ctx->revs->pending);\n }\n \n-static void do_traverse(struct rev_info *revs,\n-\t\t\tshow_commit_fn show_commit,\n-\t\t\tshow_object_fn show_object,\n-\t\t\tvoid *show_data,\n-\t\t\tfilter_object_fn filter_fn,\n-\t\t\tvoid *filter_data)\n+static void do_traverse(struct traversal_context *ctx)\n {\n \tstruct commit *commit;\n \tstruct strbuf csp; /* callee's scratch pad */\n \tstrbuf_init(&csp, PATH_MAX);\n \n-\twhile ((commit = get_revision(revs)) != NULL) {\n+\twhile ((commit = get_revision(ctx->revs)) != NULL) {\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n \t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(revs, get_commit_tree(commit));\n-\t\tshow_commit(commit, show_data);\n+\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tctx->show_commit(commit, ctx->show_data);\n \n-\t\tif (revs->tree_blobs_in_commit_order)\n+\t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n \t\t\t * NEEDSWORK: Adding the tree and then flushing it here\n \t\t\t * needs a reallocation for each commit. Can we pass the\n \t\t\t * tree directory without allocation churn?\n \t\t\t */\n-\t\t\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t\t\t show_object, show_data,\n-\t\t\t\t\t\t filter_fn, filter_data);\n+\t\t\ttraverse_trees_and_blobs(ctx, &csp);\n \t}\n-\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t show_object, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\ttraverse_trees_and_blobs(ctx, &csp);\n \tstrbuf_release(&csp);\n }\n \n@@ -326,7 +305,14 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t\t show_object_fn show_object,\n \t\t\t void *show_data)\n {\n-\tdo_traverse(revs, show_commit, show_object, show_data, NULL, NULL);\n+\tstruct traversal_context ctx;\n+\tctx.revs = revs;\n+\tctx.show_commit = show_commit;\n+\tctx.show_object = show_object;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\tctx.filter_data = NULL;\n+\tdo_traverse(&ctx);\n }\n \n void traverse_commit_list_filtered(\n@@ -337,14 +323,18 @@ void traverse_commit_list_filtered(\n \tvoid *show_data,\n \tstruct oidset *omitted)\n {\n-\tfilter_object_fn filter_fn = NULL;\n+\tstruct traversal_context ctx;\n \tfilter_free_fn filter_free_fn = NULL;\n-\tvoid *filter_data = NULL;\n-\n-\tfilter_data = list_objects_filter__init(omitted, filter_options,\n-\t\t\t\t\t\t&filter_fn, &filter_free_fn);\n-\tdo_traverse(revs, show_commit, show_object, show_data,\n-\t\t filter_fn, filter_data);\n-\tif (filter_data && filter_free_fn)\n-\t\tfilter_free_fn(filter_data);\n+\n+\tctx.revs = revs;\n+\tctx.show_object = show_object;\n+\tctx.show_commit = show_commit;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\n+\tctx.filter_data = list_objects_filter__init(omitted, filter_options,\n+\t\t\t\t\t\t &ctx.filter_fn, &filter_free_fn);\n+\tdo_traverse(&ctx);\n+\tif (ctx.filter_data && filter_free_fn)\n+\t\tfilter_free_fn(ctx.filter_data);\n }\n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355211","messageId":"feb7ea58228df4bb777ee96651389000e804f9fa.1533941986.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533941986.git.matvore@google.com","subject":"[PATCH v2 2/5] list-objects: refactor to process_tree_contents","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-10T23:06:26Z","receivedAt":"2018-08-10T23:06:55Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will be used in a follow-up patch to reduce indentation needed when\ninvoking the logic conditionally. i.e. rather than:\n\nif (foo) {\n\twhile (...) {\n\t\t/* this is very indented */\n\t}\n}\n\nwe will have:\n\nif (foo)\n\tprocess_tree_contents(...);\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 68 ++++++++++++++++++++++++++++++--------------------\n 1 file changed, 41 insertions(+), 27 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 584518a3f..ccc529e5e 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -94,6 +94,46 @@ static void process_gitlink(struct traversal_context *ctx,\n \t/* Nothing to do */\n }\n \n+static void process_tree(struct traversal_context *ctx,\n+\t\t\t struct tree *tree,\n+\t\t\t struct strbuf *base,\n+\t\t\t const char *name);\n+\n+static void process_tree_contents(struct traversal_context *ctx,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *base)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tenum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ?\n+\t\tall_entries_interesting : entry_not_interesting;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tif (match != all_entries_interesting) {\n+\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n+\t\t\t\t\t\t &ctx->revs->diffopt.pathspec);\n+\t\t\tif (match == all_entries_not_interesting)\n+\t\t\t\tbreak;\n+\t\t\tif (match == entry_not_interesting)\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (S_ISDIR(entry.mode))\n+\t\t\tprocess_tree(ctx,\n+\t\t\t\t lookup_tree(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t\telse if (S_ISGITLINK(entry.mode))\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash,\n+\t\t\t\t\tbase, entry.path);\n+\t\telse\n+\t\t\tprocess_blob(ctx,\n+\t\t\t\t lookup_blob(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t}\n+}\n+\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n@@ -101,10 +141,6 @@ static void process_tree(struct traversal_context *ctx,\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n-\tstruct tree_desc desc;\n-\tstruct name_entry entry;\n-\tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n-\t\tall_entries_interesting: entry_not_interesting;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \tint gently = revs->ignore_missing_links ||\n@@ -144,29 +180,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tinit_tree_desc(&desc, tree->buffer, tree->size);\n-\n-\twhile (tree_entry(&desc, &entry)) {\n-\t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n-\t\t\t\t\t\t &revs->diffopt.pathspec);\n-\t\t\tif (match == all_entries_not_interesting)\n-\t\t\t\tbreak;\n-\t\t\tif (match == entry_not_interesting)\n-\t\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t}\n+\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355212","messageId":"6d6b06e870b5d6dbe9bb88bd80fcce34ba4dcccf.1533941986.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533941986.git.matvore@google.com","subject":"[PATCH v2 3/5] rev-list: handle missing tree objects properly","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-10T23:06:27Z","receivedAt":"2018-08-10T23:06:58Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Previously, we assumed only blob objects could be missing. This patch\nmakes rev-list handle missing trees like missing blobs. A missing tree\nwill cause an error if --missing indicates an error should be caused,\nand the hash is printed even if the tree is missing.\n\nIn list-objects.c we no longer print a message to stderr if a tree\nobject is missing (quiet_on_missing is always true). I couldn't find\nany place where this would matter, or where the caller of\ntraverse_commit_list would need to be fixed to show the error. However,\nin the future it would be trivial to make the caller show the message if\nwe needed to.\n\nThis is not tested very thoroughly, since we cannot create promisor\nobjects in tests without using an actual partial clone. t0410 has a\npromise_and_delete utility function, but the is_promisor_object function\ndoes not return 1 for objects deleted in this way. More tests will will\ncome in a patch that implements a filter that can be used with git\nclone.\n\nSigned-off-by: Matthew DeVore \n---\n builtin/rev-list.c | 10 ++++++----\n list-objects.c | 17 +++++++++--------\n t/t5317-pack-objects-filter-objects.sh | 13 +++++++++++++\n 3 files changed, 28 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 5b07f3f4a..ea0daf0c4 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -6,6 +6,7 @@\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"object.h\"\n #include \"object-store.h\"\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n@@ -209,7 +210,8 @@ static inline void finish_object__ma(struct object *obj)\n \t */\n \tswitch (arg_missing_action) {\n \tcase MA_ERROR:\n-\t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n+\t\tdie(\"missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tcase MA_ALLOW_ANY:\n@@ -222,8 +224,8 @@ static inline void finish_object__ma(struct object *obj)\n \tcase MA_ALLOW_PROMISOR:\n \t\tif (is_promisor_object(&obj->oid))\n \t\t\treturn;\n-\t\tdie(\"unexpected missing blob object '%s'\",\n-\t\t oid_to_hex(&obj->oid));\n+\t\tdie(\"unexpected missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tdefault:\n@@ -235,7 +237,7 @@ static inline void finish_object__ma(struct object *obj)\n static int finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\tif (!has_object_file(&obj->oid)) {\n \t\tfinish_object__ma(obj);\n \t\treturn 1;\n \t}\ndiff --git a/list-objects.c b/list-objects.c\nindex ccc529e5e..aedcd0228 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,8 +143,7 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n-\tint gently = revs->ignore_missing_links ||\n-\t\t revs->exclude_promisor_objects;\n+\tint parsed;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -152,20 +151,21 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, gently) < 0) {\n+\tparsed = parse_tree_gently(tree, /*quiet_on_missing=*/1) >= 0;\n+\tif (!parsed) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n+\t\tif (!is_promisor_object(&obj->oid))\n+\t\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n+\n \t\t/*\n \t\t * Pre-filter known-missing tree objects when explicitly\n \t\t * requested. This may cause the actual filter to report\n \t\t * an incomplete list of missing objects.\n \t\t */\n-\t\tif (revs->exclude_promisor_objects &&\n-\t\t is_promisor_object(&obj->oid))\n+\t\tif (revs->exclude_promisor_objects)\n \t\t\treturn;\n-\n-\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \n \tstrbuf_addstr(base, name);\n@@ -180,7 +180,8 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tprocess_tree_contents(ctx, tree, base);\n+\tif (parsed)\n+\t\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 6710c8bc8..5e35f33bf 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -59,6 +59,19 @@ test_expect_success 'verify normal and blob:none packfiles have same commits/tre\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'get an error for missing tree object' '\n+\tgit init r5 &&\n+\techo foo > r5/foo &&\n+\tgit -C r5 add foo &&\n+\tgit -C r5 commit -m \"foo\" &&\n+\tdel=$(git -C r5 rev-parse HEAD^{tree} | sed \"s|..|&/|\") &&\n+\trm r5/.git/objects/$del &&\n+\ttest_must_fail git -C r5 pack-objects --rev --stdout 2>bad_tree <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgrep -q \"bad tree object\" bad_tree\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355213","messageId":"448e2dc86aa009c1a0872739fa550c3f97d7bed1.1533941986.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533941986.git.matvore@google.com","subject":"[PATCH v2 4/5] revision: mark non-user-given objects instead","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-10T23:06:28Z","receivedAt":"2018-08-10T23:07:01Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Currently, list-objects.c incorrectly treats all root trees of commits\nas USER_GIVEN. Also, it would be easier to mark objects that are\nnon-user-given instead of user-given, since the places in the code\nwhere we access an object through a reference are more obvious than\nthe places where we access an object that was given by the user.\n\nResolve these two problems by introducing a flag NOT_USER_GIVEN that\nmarks blobs and trees that are non-user-given, replacing USER_GIVEN.\n(Only blobs and trees are marked because this mark is only used when\nfiltering objects, and filtering of other types of objects is not\nsupported yet.)\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 31 ++++++++++++++++++-------------\n revision.c | 1 -\n revision.h | 10 +++++++---\n 3 files changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex aedcd0228..fd522a59a 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -53,7 +53,7 @@ static void process_blob(struct traversal_context *ctx,\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n \t\t\t\t path->buf, &path->buf[pathlen],\n \t\t\t\t ctx->filter_data);\n@@ -120,17 +120,19 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\tcontinue;\n \t\t}\n \n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\tstruct tree *t = lookup_tree(the_repository, entry.oid);\n+\t\t\tt->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_tree(ctx, t, base, entry.path);\n+\t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid->hash,\n \t\t\t\t\tbase, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\telse {\n+\t\t\tstruct blob *b = lookup_blob(the_repository, entry.oid);\n+\t\t\tb->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_blob(ctx, b, base, entry.path);\n+\t\t}\n \t}\n }\n \n@@ -169,7 +171,7 @@ static void process_tree(struct traversal_context *ctx,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -183,7 +185,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (parsed)\n \t\tprocess_tree_contents(ctx, tree, base);\n \n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -299,8 +301,11 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n-\t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tif (get_commit_tree(commit)) {\n+\t\t\tstruct tree *tree = get_commit_tree(commit);\n+\t\t\ttree->object.flags |= NOT_USER_GIVEN;\n+\t\t\tadd_pending_tree(ctx->revs, tree);\n+\t\t}\n \t\tctx->show_commit(commit, ctx->show_data);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\ndiff --git a/revision.c b/revision.c\nindex 062749437..6d355b43c 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -175,7 +175,6 @@ static void add_pending_object_with_path(struct rev_info *revs,\n \t\tstrbuf_release(&buf);\n \t\treturn; /* do not add the commit itself */\n \t}\n-\tobj->flags |= USER_GIVEN;\n \tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n }\n \ndiff --git a/revision.h b/revision.h\nindex c599c34da..cd6b62313 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -8,7 +8,11 @@\n #include \"diff.h\"\n #include \"commit-slab-decl.h\"\n \n-/* Remember to update object flag allocation in object.h */\n+/* Remember to update object flag allocation in object.h\n+ * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n+ * filtering trees and blobs, but it may be useful to support filtering commits\n+ * in the future.\n+ */\n #define SEEN\t\t(1u<<0)\n #define UNINTERESTING (1u<<1)\n #define TREESAME\t(1u<<2)\n@@ -20,9 +24,9 @@\n #define SYMMETRIC_LEFT\t(1u<<8)\n #define PATCHSAME\t(1u<<9)\n #define BOTTOM\t\t(1u<<10)\n-#define USER_GIVEN\t(1u<<25) /* given directly by the user */\n+#define NOT_USER_GIVEN\t(1u<<25) /* tree or blob not given directly by user */\n #define TRACK_LINEAR\t(1u<<26)\n-#define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n+#define ALL_REV_FLAGS\t(((1u<<11)-1) | NOT_USER_GIVEN | TRACK_LINEAR)\n \n #define DECORATE_SHORT_REFS\t1\n #define DECORATE_FULL_REFS\t2\n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355214","messageId":"0af4ef64aa52d76a597b4e97c0626c70fb2a883b.1533941986.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533941986.git.matvore@google.com","subject":"[PATCH v2 5/5] list-objects-filter: implement filter tree:none","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-10T23:06:29Z","receivedAt":"2018-08-10T23:07:03Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Teach list-objects the \"tree:none\" filter which allows for filtering\nout all tree and blob objects (unless other objects are explicitly\nspecified by the user). The purpose of this patch is to allow smaller\npartial clones.\n\nThe name of this filter - tree:none - does not explicitly specify that\nit also filters out all blobs, but this should not cause much confusion\nbecause blobs are not at all useful without the trees that refer to\nthem.\n\nI also consider only:commits as a name, but this is inaccurate because\nit suggests that annotated tags are omitted, but actually they are\nincluded.\n\nSigned-off-by: Matthew DeVore \n---\n Documentation/rev-list-options.txt | 2 ++\n list-objects-filter-options.c | 4 +++\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 49 +++++++++++++++++++-------\n t/t5317-pack-objects-filter-objects.sh | 27 ++++++++++++++\n t/t5616-partial-clone.sh | 27 ++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 13 +++++++\n 7 files changed, 110 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 7b273635d..68b4b9552 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -743,6 +743,8 @@ specification contained in .\n \tA debug option to help with future \"partial clone\" development.\n \tThis option specifies how missing objects are handled.\n +\n+The form '--filter=tree:none' omits all blobs and trees.\n++\n The form '--missing=error' requests that rev-list stop with an error if\n a missing object is encountered. This is the default action.\n +\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex c0e2bd6a0..523cb00a0 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -50,6 +50,10 @@ static int gently_parse_list_objects_filter(\n \t\t\treturn 0;\n \t\t}\n \n+\t} else if (!strcmp(arg, \"tree:none\")) {\n+\t\tfilter_options->choice = LOFC_TREE_NONE;\n+\t\treturn 0;\n+\n \t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n \t\tstruct object_context oc;\n \t\tstruct object_id sparse_oid;\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 0000a61f8..af64e5c66 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -10,6 +10,7 @@ enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n+\tLOFC_TREE_NONE,\n \tLOFC_SPARSE_OID,\n \tLOFC_SPARSE_PATH,\n \tLOFC__COUNT /* must be last */\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex a0ba78b20..22c894093 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -26,38 +26,45 @@\n #define FILTER_SHOWN_BUT_REVISIT (1<<21)\n \n /*\n- * A filter for list-objects to omit ALL blobs from the traversal.\n- * And to OPTIONALLY collect a list of the omitted OIDs.\n+ * A filter for list-objects to omit ALL blobs from the traversal, and possibly\n+ * trees as well.\n+ * Can OPTIONALLY collect a list of the omitted OIDs.\n */\n-struct filter_blobs_none_data {\n+struct filter_none_of_type_data {\n+\t/* blobs are always omitted */\n+\tunsigned omit_trees : 1;\n \tstruct oidset *omits;\n };\n \n-static enum list_objects_filter_result filter_blobs_none(\n+static enum list_objects_filter_result filter_none_of_type(\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n \tvoid *filter_data_)\n {\n-\tstruct filter_blobs_none_data *filter_data = filter_data_;\n+\tstruct filter_none_of_type_data *filter_data = filter_data_;\n \n \tswitch (filter_situation) {\n \tdefault:\n \t\tdie(\"unknown filter_situation\");\n \t\treturn LOFR_ZERO;\n \n-\tcase LOFS_BEGIN_TREE:\n-\t\tassert(obj->type == OBJ_TREE);\n-\t\t/* always include all tree objects */\n-\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n-\n \tcase LOFS_END_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\treturn LOFR_ZERO;\n \n+\tcase LOFS_BEGIN_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\tif (!filter_data->omit_trees)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\t\t/*\n+\t\t * Fallthrough to insert into omitted list for trees as well as\n+\t\t * blobs.\n+\t\t */\n+\t\t/* fallthrough */\n \tcase LOFS_BLOB:\n-\t\tassert(obj->type == OBJ_BLOB);\n \t\tassert((obj->flags & SEEN) == 0);\n \n \t\tif (filter_data->omits)\n@@ -72,10 +79,25 @@ static void *filter_blobs_none__init(\n \tfilter_object_fn *filter_fn,\n \tfilter_free_fn *filter_free_fn)\n {\n-\tstruct filter_blobs_none_data *d = xcalloc(1, sizeof(*d));\n+\tstruct filter_none_of_type_data *d = xcalloc(1, sizeof(*d));\n+\td->omits = omitted;\n+\n+\t*filter_fn = filter_none_of_type;\n+\t*filter_free_fn = free;\n+\treturn d;\n+}\n+\n+static void* filter_tree_none__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn)\n+{\n+\tstruct filter_none_of_type_data *d = xcalloc(1, sizeof(*d));\n+\td->omit_trees = 1;\n \td->omits = omitted;\n \n-\t*filter_fn = filter_blobs_none;\n+\t*filter_fn = filter_none_of_type;\n \t*filter_free_fn = free;\n \treturn d;\n }\n@@ -374,6 +396,7 @@ static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n+\tfilter_tree_none__init,\n \tfilter_sparse_oid__init,\n \tfilter_sparse_path__init,\n };\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 5e35f33bf..28a8c916a 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -72,6 +72,33 @@ test_expect_success 'get an error for missing tree object' '\n \tgrep -q \"bad tree object\" bad_tree\n '\n \n+test_expect_success 'setup for tests of tree:none' '\n+\tmkdir r1/subtree &&\n+\techo \"This is a file in a subtree\" > r1/subtree/file &&\n+\tgit -C r1 add subtree/file &&\n+\tgit -C r1 commit -m subtree\n+'\n+\n+test_expect_success 'verify tree:none packfile has no blobs or trees' '\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:none >commitsonly.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\t! grep -E \"tree|blob\" objs\n+'\n+\n+test_expect_success 'grab tree directly when using tree:none' '\n+\t# We should get the tree specified directly but not its blobs or subtrees.\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:none >commitsonly.pack <<-EOF &&\n+\tHEAD:\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\tgrep -E \"tree|blob\" objs >trees_and_blobs &&\n+\ttest_line_count = 1 trees_and_blobs\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex bbbe7537d..4fc068716 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -170,6 +170,33 @@ test_expect_success 'partial clone fetches blobs pointed to by refs even if norm\n \tgit -C dst fsck\n '\n \n+test_expect_success 'can use tree:none to filter partial clone' '\n+\trm -rf dst &&\n+\tgit clone --no-checkout --filter=tree:none \"file://$(pwd)/srv.bare\" dst &&\n+\tgit -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n+\tcat fetched_objects \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n+\tsort fetched_types -u >unique_types.observed &&\n+\techo commit > unique_types.expected &&\n+\ttest_cmp unique_types.observed unique_types.expected\n+'\n+\n+test_expect_success 'show missing tree objects with --missing=print' '\n+\tgit -C dst rev-list master --missing=print --quiet --objects >missing_objs &&\n+\tsed \"s/?//\" missing_objs \\\n+\t\t| xargs -n1 git -C srv.bare cat-file -t \\\n+\t\t>missing_types &&\n+\tsort -u missing_types >missing_types.uniq &&\n+\techo tree >expected &&\n+\ttest_cmp missing_types.uniq expected\n+'\n+\n+test_expect_success 'do not complain when a missing tree cannot be parsed' '\n+\tgit -C dst rev-list master --missing=print --quiet --objects 2>rev_list_err >&2 &&\n+\t! grep -q \"Could not read \" rev_list_err\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 0a37dd5f9..ecdf6b4c3 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -196,6 +196,19 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n \ttest_cmp observed expected\n '\n \n+# Test tree:none filter.\n+\n+test_expect_success 'verify tree:none includes trees in \"filtered\" output' '\n+\tgit -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=tree:none \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed s/~// \\\n+\t\t| xargs -n1 git -C r3 cat-file -t \\\n+\t\t| sort -u >filtered_types &&\n+\tprintf \"blob\\ntree\\n\" > expected &&\n+\ttest_cmp filtered_types expected\n+'\n+\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355396","messageId":"ce743b3b-9d60-a6d7-9064-e9272c875e00@jeffhostetler.com","threadId":"49086","inReplyTo":"0af4ef64aa52d76a597b4e97c0626c70fb2a883b.1533941986.git.matvore@google.com","subject":"Re: [PATCH v2 5/5] list-objects-filter: implement filter tree:none","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2018-08-13T16:38:57Z","receivedAt":"2018-08-13T16:39:02Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 8/10/2018 7:06 PM, Matthew DeVore wrote:\n> Teach list-objects the \"tree:none\" filter which allows for filtering\n> out all tree and blob objects (unless other objects are explicitly\n> specified by the user). The purpose of this patch is to allow smaller\n> partial clones.\n> \n> The name of this filter - tree:none - does not explicitly specify that\n> it also filters out all blobs, but this should not cause much confusion\n> because blobs are not at all useful without the trees that refer to\n> them.\n> \n> I also consider only:commits as a name, but this is inaccurate because\n> it suggests that annotated tags are omitted, but actually they are\n> included.\n> \n> Signed-off-by: Matthew DeVore \n> ---\n> Documentation/rev-list-options.txt | 2 ++\n> list-objects-filter-options.c | 4 +++\n> list-objects-filter-options.h | 1 +\n> list-objects-filter.c | 49 +++++++++++++++++++-------\n> t/t5317-pack-objects-filter-objects.sh | 27 ++++++++++++++\n> t/t5616-partial-clone.sh | 27 ++++++++++++++\n> t/t6112-rev-list-filters-objects.sh | 13 +++++++\n> 7 files changed, 110 insertions(+), 13 deletions(-)\n> \n> diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\n> index 7b273635d..68b4b9552 100644\n> --- a/Documentation/rev-list-options.txt\n> +++ b/Documentation/rev-list-options.txt\n> @@ -743,6 +743,8 @@ specification contained in .\n> \tA debug option to help with future \"partial clone\" development.\n> \tThis option specifies how missing objects are handled.\n> +\n> +The form '--filter=tree:none' omits all blobs and trees.\n> ++\n> The form '--missing=error' requests that rev-list stop with an error if\n> a missing object is encountered. This is the default action.\n> +\n> diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\n> index c0e2bd6a0..523cb00a0 100644\n> --- a/list-objects-filter-options.c\n> +++ b/list-objects-filter-options.c\n> @@ -50,6 +50,10 @@ static int gently_parse_list_objects_filter(\n> \t\t\treturn 0;\n> \t\t}\n> \n> +\t} else if (!strcmp(arg, \"tree:none\")) {\n> +\t\tfilter_options->choice = LOFC_TREE_NONE;\n> +\t\treturn 0;\n> +\n> \t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n> \t\tstruct object_context oc;\n> \t\tstruct object_id sparse_oid;\n> diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\n> index 0000a61f8..af64e5c66 100644\n> --- a/list-objects-filter-options.h\n> +++ b/list-objects-filter-options.h\n> @@ -10,6 +10,7 @@ enum list_objects_filter_choice {\n> \tLOFC_DISABLED = 0,\n> \tLOFC_BLOB_NONE,\n> \tLOFC_BLOB_LIMIT,\n> +\tLOFC_TREE_NONE,\n> \tLOFC_SPARSE_OID,\n> \tLOFC_SPARSE_PATH,\n> \tLOFC__COUNT /* must be last */\n> diff --git a/list-objects-filter.c b/list-objects-filter.c\n> index a0ba78b20..22c894093 100644\n> --- a/list-objects-filter.c\n> +++ b/list-objects-filter.c\n> @@ -26,38 +26,45 @@\n> #define FILTER_SHOWN_BUT_REVISIT (1<<21)\n> \n> /*\n> - * A filter for list-objects to omit ALL blobs from the traversal.\n> - * And to OPTIONALLY collect a list of the omitted OIDs.\n> + * A filter for list-objects to omit ALL blobs from the traversal, and possibly\n> + * trees as well.\n> + * Can OPTIONALLY collect a list of the omitted OIDs.\n> */\n> -struct filter_blobs_none_data {\n> +struct filter_none_of_type_data {\n> +\t/* blobs are always omitted */\n> +\tunsigned omit_trees : 1;\n> \tstruct oidset *omits;\n> };\n> \n\nI'm not sure I'd convert the existing filter types.\nWhen I created this file, I created a set of function pairs\nfor each filter type:\n filter_() and filter___init()\n\nwith the latter being added to the s_filters[] array and created\na choice enum having corresponding values\n LOFC_\n\nHere you're adding a new _init() and LOFC_ key, but mapping both\nthe original \"blob:none\" and the new \"tree:none\" to a combined\nfilter function and blends these 2 modes.\n\nStyle-wise, I'd keep the original filters as they were and add a\nnew function pair for the new tree:none filter. Then you can\nsimplify the logic inside your new filter. For example, in your\nfilter \"filter_data->omit_trees\" will always be true, so you can\njust do the \"if (filter_data->omits) oidset_insert(...); return _SEEN\"\nand not have the fallthru stuff -- or get rid of the asserts() and put\nthe case labels together.\n\nOne of the things I wanted to do (when I found some free time) was to\nadd a \"tree:none\" and maybe a \"tree:root\" filter. (The latter only\nincluding the root trees associated with the fetched commits, since\nthere are/were some places where we implicitly also load the root tree\nwhen loading the commit object.) So in that vein, it might be that we\nwould want a \"tree:\" filter instead with 0 = none and 1 = root.\nI wasn't ready to propose that when I did the filtering, but I had that\nin mind. (And is partially why I suggest keeping your new filter\nindependent of the existing ones.)\n\nJeff\n\n\n> -static enum list_objects_filter_result filter_blobs_none(\n> +static enum list_objects_filter_result filter_none_of_type(\n> \tenum list_objects_filter_situation filter_situation,\n> \tstruct object *obj,\n> \tconst char *pathname,\n> \tconst char *filename,\n> \tvoid *filter_data_)\n> {\n> -\tstruct filter_blobs_none_data *filter_data = filter_data_;\n> +\tstruct filter_none_of_type_data *filter_data = filter_data_;\n> \n> \tswitch (filter_situation) {\n> \tdefault:\n> \t\tdie(\"unknown filter_situation\");\n> \t\treturn LOFR_ZERO;\n> \n> -\tcase LOFS_BEGIN_TREE:\n> -\t\tassert(obj->type == OBJ_TREE);\n> -\t\t/* always include all tree objects */\n> -\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> -\n> \tcase LOFS_END_TREE:\n> \t\tassert(obj->type == OBJ_TREE);\n> \t\treturn LOFR_ZERO;\n> \n> +\tcase LOFS_BEGIN_TREE:\n> +\t\tassert(obj->type == OBJ_TREE);\n> +\t\tif (!filter_data->omit_trees)\n> +\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> +\n> +\t\t/*\n> +\t\t * Fallthrough to insert into omitted list for trees as well as\n> +\t\t * blobs.\n> +\t\t */\n> +\t\t/* fallthrough */\n> \tcase LOFS_BLOB:\n> -\t\tassert(obj->type == OBJ_BLOB);\n> \t\tassert((obj->flags & SEEN) == 0);\n> \n> \t\tif (filter_data->omits)\n> @@ -72,10 +79,25 @@ static void *filter_blobs_none__init(\n> \tfilter_object_fn *filter_fn,\n> \tfilter_free_fn *filter_free_fn)\n> {\n> -\tstruct filter_blobs_none_data *d = xcalloc(1, sizeof(*d));\n> +\tstruct filter_none_of_type_data *d = xcalloc(1, sizeof(*d));\n> +\td->omits = omitted;\n> +\n> +\t*filter_fn = filter_none_of_type;\n> +\t*filter_free_fn = free;\n> +\treturn d;\n> +}\n> +\n> +static void* filter_tree_none__init(\n> +\tstruct oidset *omitted,\n> +\tstruct list_objects_filter_options *filter_options,\n> +\tfilter_object_fn *filter_fn,\n> +\tfilter_free_fn *filter_free_fn)\n> +{\n> +\tstruct filter_none_of_type_data *d = xcalloc(1, sizeof(*d));\n> +\td->omit_trees = 1;\n> \td->omits = omitted;\n> \n> -\t*filter_fn = filter_blobs_none;\n> +\t*filter_fn = filter_none_of_type;\n> \t*filter_free_fn = free;\n> \treturn d;\n> }\n> @@ -374,6 +396,7 @@ static filter_init_fn s_filters[] = {\n> \tNULL,\n> \tfilter_blobs_none__init,\n> \tfilter_blobs_limit__init,\n> +\tfilter_tree_none__init,\n> \tfilter_sparse_oid__init,\n> \tfilter_sparse_path__init,\n> };\n> diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\n> index 5e35f33bf..28a8c916a 100755\n> --- a/t/t5317-pack-objects-filter-objects.sh\n> +++ b/t/t5317-pack-objects-filter-objects.sh\n> @@ -72,6 +72,33 @@ test_expect_success 'get an error for missing tree object' '\n> \tgrep -q \"bad tree object\" bad_tree\n> '\n> \n> +test_expect_success 'setup for tests of tree:none' '\n> +\tmkdir r1/subtree &&\n> +\techo \"This is a file in a subtree\" > r1/subtree/file &&\n> +\tgit -C r1 add subtree/file &&\n> +\tgit -C r1 commit -m subtree\n> +'\n> +\n> +test_expect_success 'verify tree:none packfile has no blobs or trees' '\n> +\tgit -C r1 pack-objects --rev --stdout --filter=tree:none >commitsonly.pack <<-EOF &&\n> +\tHEAD\n> +\tEOF\n> +\tgit -C r1 index-pack ../commitsonly.pack &&\n> +\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n> +\t! grep -E \"tree|blob\" objs\n> +'\n> +\n> +test_expect_success 'grab tree directly when using tree:none' '\n> +\t# We should get the tree specified directly but not its blobs or subtrees.\n> +\tgit -C r1 pack-objects --rev --stdout --filter=tree:none >commitsonly.pack <<-EOF &&\n> +\tHEAD:\n> +\tEOF\n> +\tgit -C r1 index-pack ../commitsonly.pack &&\n> +\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n> +\tgrep -E \"tree|blob\" objs >trees_and_blobs &&\n> +\ttest_line_count = 1 trees_and_blobs\n> +'\n> +\n> # Test blob:limit=[kmg] filter.\n> # We boundary test around the size parameter. The filter is strictly less than\n> # the value, so size 500 and 1000 should have the same results, but 1001 should\n> diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\n> index bbbe7537d..4fc068716 100755\n> --- a/t/t5616-partial-clone.sh\n> +++ b/t/t5616-partial-clone.sh\n> @@ -170,6 +170,33 @@ test_expect_success 'partial clone fetches blobs pointed to by refs even if norm\n> \tgit -C dst fsck\n> '\n> \n> +test_expect_success 'can use tree:none to filter partial clone' '\n> +\trm -rf dst &&\n> +\tgit clone --no-checkout --filter=tree:none \"file://$(pwd)/srv.bare\" dst &&\n> +\tgit -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n> +\tcat fetched_objects \\\n> +\t\t| awk -f print_1.awk \\\n> +\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n> +\tsort fetched_types -u >unique_types.observed &&\n> +\techo commit > unique_types.expected &&\n> +\ttest_cmp unique_types.observed unique_types.expected\n> +'\n> +\n> +test_expect_success 'show missing tree objects with --missing=print' '\n> +\tgit -C dst rev-list master --missing=print --quiet --objects >missing_objs &&\n> +\tsed \"s/?//\" missing_objs \\\n> +\t\t| xargs -n1 git -C srv.bare cat-file -t \\\n> +\t\t>missing_types &&\n> +\tsort -u missing_types >missing_types.uniq &&\n> +\techo tree >expected &&\n> +\ttest_cmp missing_types.uniq expected\n> +'\n> +\n> +test_expect_success 'do not complain when a missing tree cannot be parsed' '\n> +\tgit -C dst rev-list master --missing=print --quiet --objects 2>rev_list_err >&2 &&\n> +\t! grep -q \"Could not read \" rev_list_err\n> +'\n> +\n> . \"$TEST_DIRECTORY\"/lib-httpd.sh\n> start_httpd\n> \n> diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\n> index 0a37dd5f9..ecdf6b4c3 100755\n> --- a/t/t6112-rev-list-filters-objects.sh\n> +++ b/t/t6112-rev-list-filters-objects.sh\n> @@ -196,6 +196,19 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n> \ttest_cmp observed expected\n> '\n> \n> +# Test tree:none filter.\n> +\n> +test_expect_success 'verify tree:none includes trees in \"filtered\" output' '\n> +\tgit -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=tree:none \\\n> +\t\t| awk -f print_1.awk \\\n> +\t\t| sed s/~// \\\n> +\t\t| xargs -n1 git -C r3 cat-file -t \\\n> +\t\t| sort -u >filtered_types &&\n> +\tprintf \"blob\\ntree\\n\" > expected &&\n> +\ttest_cmp filtered_types expected\n> +'\n> +\n> +\n> # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n> # This models previously omitted objects that we did not receive.\n> \n> \n"},{"id":"355423","messageId":"cover.1534183648.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"[PATCH v3 0/5] filter: support for excluding all trees and blobs","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-13T18:14:27Z","receivedAt":"2018-08-13T18:14:52Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Applied the following changes suggested by git@jeffhostetler.com:\n - Change the filter name from tree:none to tree:0 to make room for\n future improvements which allow filtering based on depth.\n - Made a separate filter logic function and filter data struct for\n tree:0 rather than share it with blob:none. I would usually prefer\n making the code less redundant whenever possible, but since there\n are plans to extend this in the future, it makes more sense to have\n this separate.\n\nMatthew DeVore (5):\n list-objects: store common func args in struct\n list-objects: refactor to process_tree_contents\n rev-list: handle missing tree objects properly\n revision: mark non-user-given objects instead\n list-objects-filter: implement filter tree:0\n\n Documentation/rev-list-options.txt | 3 +\n builtin/rev-list.c | 10 +-\n list-objects-filter-options.c | 4 +\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 50 ++++++\n list-objects.c | 236 +++++++++++++------------\n revision.c | 1 -\n revision.h | 10 +-\n t/t5317-pack-objects-filter-objects.sh | 40 +++++\n t/t5616-partial-clone.sh | 27 +++\n t/t6112-rev-list-filters-objects.sh | 13 ++\n 11 files changed, 274 insertions(+), 121 deletions(-)\n\n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355424","messageId":"c54a2818125739b7dde40aeee2cf1d72e057e963.1534183648.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534183648.git.matvore@google.com","subject":"[PATCH v3 1/5] list-objects: store common func args in struct","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-13T18:14:28Z","receivedAt":"2018-08-13T18:14:54Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will make utility functions easier to create, as done by the next\npatch.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 158 +++++++++++++++++++++++--------------------------\n 1 file changed, 74 insertions(+), 84 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex c99c47ac1..584518a3f 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -12,20 +12,25 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n \n-static void process_blob(struct rev_info *revs,\n+struct traversal_context {\n+\tstruct rev_info *revs;\n+\tshow_object_fn show_object;\n+\tshow_commit_fn show_commit;\n+\tvoid *show_data;\n+\tfilter_object_fn filter_fn;\n+\tvoid *filter_data;\n+};\n+\n+static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \n-\tif (!revs->blob_objects)\n+\tif (!ctx->revs->blob_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad blob object\");\n@@ -41,21 +46,21 @@ static void process_blob(struct rev_info *revs,\n \t * may cause the actual filter to report an incomplete list\n \t * of missing objects.\n \t */\n-\tif (revs->exclude_promisor_objects &&\n+\tif (ctx->revs->exclude_promisor_objects &&\n \t !has_object_file(&obj->oid) &&\n \t is_promisor_object(&obj->oid))\n \t\treturn;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BLOB, obj,\n-\t\t\t path->buf, &path->buf[pathlen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n+\t\t\t\t path->buf, &path->buf[pathlen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, path->buf, cb_data);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -81,26 +86,21 @@ static void process_blob(struct rev_info *revs,\n * the link, and how to do it. Whether it necessarily makes\n * any sense what-so-ever to ever do that is another issue.\n */\n-static void process_gitlink(struct rev_info *revs,\n+static void process_gitlink(struct traversal_context *ctx,\n \t\t\t const unsigned char *sha1,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data)\n+\t\t\t const char *name)\n {\n \t/* Nothing to do */\n }\n \n-static void process_tree(struct rev_info *revs,\n+static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &tree->object;\n+\tstruct rev_info *revs = ctx->revs;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n@@ -133,14 +133,14 @@ static void process_tree(struct rev_info *revs,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BEGIN_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, base->buf, cb_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -157,29 +157,25 @@ static void process_tree(struct rev_info *revs,\n \t\t}\n \n \t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(revs,\n+\t\t\tprocess_tree(ctx,\n \t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(revs, entry.oid->hash,\n-\t\t\t\t\tshow, base, entry.path,\n-\t\t\t\t\tcb_data);\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n \t\telse\n-\t\t\tprocess_blob(revs,\n+\t\t\tprocess_blob(ctx,\n \t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t}\n \n-\tif (!(obj->flags & USER_GIVEN) && filter_fn) {\n-\t\tr = filter_fn(LOFS_END_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \t\tif (r & LOFR_MARK_SEEN)\n \t\t\tobj->flags |= SEEN;\n \t\tif (r & LOFR_DO_SHOW)\n-\t\t\tshow(obj, base->buf, cb_data);\n+\t\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \t}\n \n \tstrbuf_setlen(base, baselen);\n@@ -242,19 +238,15 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n-static void traverse_trees_and_blobs(struct rev_info *revs,\n-\t\t\t\t struct strbuf *base,\n-\t\t\t\t show_object_fn show_object,\n-\t\t\t\t void *show_data,\n-\t\t\t\t filter_object_fn filter_fn,\n-\t\t\t\t void *filter_data)\n+static void traverse_trees_and_blobs(struct traversal_context *ctx,\n+\t\t\t\t struct strbuf *base)\n {\n \tint i;\n \n \tassert(base->len == 0);\n \n-\tfor (i = 0; i < revs->pending.nr; i++) {\n-\t\tstruct object_array_entry *pending = revs->pending.objects + i;\n+\tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n+\t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n@@ -262,62 +254,49 @@ static void traverse_trees_and_blobs(struct rev_info *revs,\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tshow_object(obj, name, show_data);\n+\t\t\tctx->show_object(obj, name, ctx->show_data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(revs, (struct tree *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(revs, (struct blob *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n \t\t oid_to_hex(&obj->oid), name);\n \t}\n-\tobject_array_clear(&revs->pending);\n+\tobject_array_clear(&ctx->revs->pending);\n }\n \n-static void do_traverse(struct rev_info *revs,\n-\t\t\tshow_commit_fn show_commit,\n-\t\t\tshow_object_fn show_object,\n-\t\t\tvoid *show_data,\n-\t\t\tfilter_object_fn filter_fn,\n-\t\t\tvoid *filter_data)\n+static void do_traverse(struct traversal_context *ctx)\n {\n \tstruct commit *commit;\n \tstruct strbuf csp; /* callee's scratch pad */\n \tstrbuf_init(&csp, PATH_MAX);\n \n-\twhile ((commit = get_revision(revs)) != NULL) {\n+\twhile ((commit = get_revision(ctx->revs)) != NULL) {\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n \t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(revs, get_commit_tree(commit));\n-\t\tshow_commit(commit, show_data);\n+\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tctx->show_commit(commit, ctx->show_data);\n \n-\t\tif (revs->tree_blobs_in_commit_order)\n+\t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n \t\t\t * NEEDSWORK: Adding the tree and then flushing it here\n \t\t\t * needs a reallocation for each commit. Can we pass the\n \t\t\t * tree directory without allocation churn?\n \t\t\t */\n-\t\t\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t\t\t show_object, show_data,\n-\t\t\t\t\t\t filter_fn, filter_data);\n+\t\t\ttraverse_trees_and_blobs(ctx, &csp);\n \t}\n-\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t show_object, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\ttraverse_trees_and_blobs(ctx, &csp);\n \tstrbuf_release(&csp);\n }\n \n@@ -326,7 +305,14 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t\t show_object_fn show_object,\n \t\t\t void *show_data)\n {\n-\tdo_traverse(revs, show_commit, show_object, show_data, NULL, NULL);\n+\tstruct traversal_context ctx;\n+\tctx.revs = revs;\n+\tctx.show_commit = show_commit;\n+\tctx.show_object = show_object;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\tctx.filter_data = NULL;\n+\tdo_traverse(&ctx);\n }\n \n void traverse_commit_list_filtered(\n@@ -337,14 +323,18 @@ void traverse_commit_list_filtered(\n \tvoid *show_data,\n \tstruct oidset *omitted)\n {\n-\tfilter_object_fn filter_fn = NULL;\n+\tstruct traversal_context ctx;\n \tfilter_free_fn filter_free_fn = NULL;\n-\tvoid *filter_data = NULL;\n-\n-\tfilter_data = list_objects_filter__init(omitted, filter_options,\n-\t\t\t\t\t\t&filter_fn, &filter_free_fn);\n-\tdo_traverse(revs, show_commit, show_object, show_data,\n-\t\t filter_fn, filter_data);\n-\tif (filter_data && filter_free_fn)\n-\t\tfilter_free_fn(filter_data);\n+\n+\tctx.revs = revs;\n+\tctx.show_object = show_object;\n+\tctx.show_commit = show_commit;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\n+\tctx.filter_data = list_objects_filter__init(omitted, filter_options,\n+\t\t\t\t\t\t &ctx.filter_fn, &filter_free_fn);\n+\tdo_traverse(&ctx);\n+\tif (ctx.filter_data && filter_free_fn)\n+\t\tfilter_free_fn(ctx.filter_data);\n }\n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355425","messageId":"feb7ea58228df4bb777ee96651389000e804f9fa.1534183648.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534183648.git.matvore@google.com","subject":"[PATCH v3 2/5] list-objects: refactor to process_tree_contents","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-13T18:14:29Z","receivedAt":"2018-08-13T18:14:59Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will be used in a follow-up patch to reduce indentation needed when\ninvoking the logic conditionally. i.e. rather than:\n\nif (foo) {\n\twhile (...) {\n\t\t/* this is very indented */\n\t}\n}\n\nwe will have:\n\nif (foo)\n\tprocess_tree_contents(...);\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 68 ++++++++++++++++++++++++++++++--------------------\n 1 file changed, 41 insertions(+), 27 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 584518a3f..ccc529e5e 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -94,6 +94,46 @@ static void process_gitlink(struct traversal_context *ctx,\n \t/* Nothing to do */\n }\n \n+static void process_tree(struct traversal_context *ctx,\n+\t\t\t struct tree *tree,\n+\t\t\t struct strbuf *base,\n+\t\t\t const char *name);\n+\n+static void process_tree_contents(struct traversal_context *ctx,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *base)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tenum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ?\n+\t\tall_entries_interesting : entry_not_interesting;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tif (match != all_entries_interesting) {\n+\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n+\t\t\t\t\t\t &ctx->revs->diffopt.pathspec);\n+\t\t\tif (match == all_entries_not_interesting)\n+\t\t\t\tbreak;\n+\t\t\tif (match == entry_not_interesting)\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (S_ISDIR(entry.mode))\n+\t\t\tprocess_tree(ctx,\n+\t\t\t\t lookup_tree(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t\telse if (S_ISGITLINK(entry.mode))\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash,\n+\t\t\t\t\tbase, entry.path);\n+\t\telse\n+\t\t\tprocess_blob(ctx,\n+\t\t\t\t lookup_blob(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t}\n+}\n+\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n@@ -101,10 +141,6 @@ static void process_tree(struct traversal_context *ctx,\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n-\tstruct tree_desc desc;\n-\tstruct name_entry entry;\n-\tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n-\t\tall_entries_interesting: entry_not_interesting;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \tint gently = revs->ignore_missing_links ||\n@@ -144,29 +180,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tinit_tree_desc(&desc, tree->buffer, tree->size);\n-\n-\twhile (tree_entry(&desc, &entry)) {\n-\t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n-\t\t\t\t\t\t &revs->diffopt.pathspec);\n-\t\t\tif (match == all_entries_not_interesting)\n-\t\t\t\tbreak;\n-\t\t\tif (match == entry_not_interesting)\n-\t\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t}\n+\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355426","messageId":"6d6b06e870b5d6dbe9bb88bd80fcce34ba4dcccf.1534183648.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534183648.git.matvore@google.com","subject":"[PATCH v3 3/5] rev-list: handle missing tree objects properly","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-13T18:14:30Z","receivedAt":"2018-08-13T18:15:08Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Previously, we assumed only blob objects could be missing. This patch\nmakes rev-list handle missing trees like missing blobs. A missing tree\nwill cause an error if --missing indicates an error should be caused,\nand the hash is printed even if the tree is missing.\n\nIn list-objects.c we no longer print a message to stderr if a tree\nobject is missing (quiet_on_missing is always true). I couldn't find\nany place where this would matter, or where the caller of\ntraverse_commit_list would need to be fixed to show the error. However,\nin the future it would be trivial to make the caller show the message if\nwe needed to.\n\nThis is not tested very thoroughly, since we cannot create promisor\nobjects in tests without using an actual partial clone. t0410 has a\npromise_and_delete utility function, but the is_promisor_object function\ndoes not return 1 for objects deleted in this way. More tests will will\ncome in a patch that implements a filter that can be used with git\nclone.\n\nSigned-off-by: Matthew DeVore \n---\n builtin/rev-list.c | 10 ++++++----\n list-objects.c | 17 +++++++++--------\n t/t5317-pack-objects-filter-objects.sh | 13 +++++++++++++\n 3 files changed, 28 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 5b07f3f4a..ea0daf0c4 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -6,6 +6,7 @@\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"object.h\"\n #include \"object-store.h\"\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n@@ -209,7 +210,8 @@ static inline void finish_object__ma(struct object *obj)\n \t */\n \tswitch (arg_missing_action) {\n \tcase MA_ERROR:\n-\t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n+\t\tdie(\"missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tcase MA_ALLOW_ANY:\n@@ -222,8 +224,8 @@ static inline void finish_object__ma(struct object *obj)\n \tcase MA_ALLOW_PROMISOR:\n \t\tif (is_promisor_object(&obj->oid))\n \t\t\treturn;\n-\t\tdie(\"unexpected missing blob object '%s'\",\n-\t\t oid_to_hex(&obj->oid));\n+\t\tdie(\"unexpected missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tdefault:\n@@ -235,7 +237,7 @@ static inline void finish_object__ma(struct object *obj)\n static int finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\tif (!has_object_file(&obj->oid)) {\n \t\tfinish_object__ma(obj);\n \t\treturn 1;\n \t}\ndiff --git a/list-objects.c b/list-objects.c\nindex ccc529e5e..aedcd0228 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,8 +143,7 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n-\tint gently = revs->ignore_missing_links ||\n-\t\t revs->exclude_promisor_objects;\n+\tint parsed;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -152,20 +151,21 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, gently) < 0) {\n+\tparsed = parse_tree_gently(tree, /*quiet_on_missing=*/1) >= 0;\n+\tif (!parsed) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n+\t\tif (!is_promisor_object(&obj->oid))\n+\t\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n+\n \t\t/*\n \t\t * Pre-filter known-missing tree objects when explicitly\n \t\t * requested. This may cause the actual filter to report\n \t\t * an incomplete list of missing objects.\n \t\t */\n-\t\tif (revs->exclude_promisor_objects &&\n-\t\t is_promisor_object(&obj->oid))\n+\t\tif (revs->exclude_promisor_objects)\n \t\t\treturn;\n-\n-\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \n \tstrbuf_addstr(base, name);\n@@ -180,7 +180,8 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tprocess_tree_contents(ctx, tree, base);\n+\tif (parsed)\n+\t\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 6710c8bc8..5e35f33bf 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -59,6 +59,19 @@ test_expect_success 'verify normal and blob:none packfiles have same commits/tre\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'get an error for missing tree object' '\n+\tgit init r5 &&\n+\techo foo > r5/foo &&\n+\tgit -C r5 add foo &&\n+\tgit -C r5 commit -m \"foo\" &&\n+\tdel=$(git -C r5 rev-parse HEAD^{tree} | sed \"s|..|&/|\") &&\n+\trm r5/.git/objects/$del &&\n+\ttest_must_fail git -C r5 pack-objects --rev --stdout 2>bad_tree <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgrep -q \"bad tree object\" bad_tree\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355427","messageId":"448e2dc86aa009c1a0872739fa550c3f97d7bed1.1534183648.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534183648.git.matvore@google.com","subject":"[PATCH v3 4/5] revision: mark non-user-given objects instead","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-13T18:14:31Z","receivedAt":"2018-08-13T18:15:08Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Currently, list-objects.c incorrectly treats all root trees of commits\nas USER_GIVEN. Also, it would be easier to mark objects that are\nnon-user-given instead of user-given, since the places in the code\nwhere we access an object through a reference are more obvious than\nthe places where we access an object that was given by the user.\n\nResolve these two problems by introducing a flag NOT_USER_GIVEN that\nmarks blobs and trees that are non-user-given, replacing USER_GIVEN.\n(Only blobs and trees are marked because this mark is only used when\nfiltering objects, and filtering of other types of objects is not\nsupported yet.)\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 31 ++++++++++++++++++-------------\n revision.c | 1 -\n revision.h | 10 +++++++---\n 3 files changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex aedcd0228..fd522a59a 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -53,7 +53,7 @@ static void process_blob(struct traversal_context *ctx,\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n \t\t\t\t path->buf, &path->buf[pathlen],\n \t\t\t\t ctx->filter_data);\n@@ -120,17 +120,19 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\tcontinue;\n \t\t}\n \n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\tstruct tree *t = lookup_tree(the_repository, entry.oid);\n+\t\t\tt->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_tree(ctx, t, base, entry.path);\n+\t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid->hash,\n \t\t\t\t\tbase, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\telse {\n+\t\t\tstruct blob *b = lookup_blob(the_repository, entry.oid);\n+\t\t\tb->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_blob(ctx, b, base, entry.path);\n+\t\t}\n \t}\n }\n \n@@ -169,7 +171,7 @@ static void process_tree(struct traversal_context *ctx,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -183,7 +185,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (parsed)\n \t\tprocess_tree_contents(ctx, tree, base);\n \n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -299,8 +301,11 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n-\t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tif (get_commit_tree(commit)) {\n+\t\t\tstruct tree *tree = get_commit_tree(commit);\n+\t\t\ttree->object.flags |= NOT_USER_GIVEN;\n+\t\t\tadd_pending_tree(ctx->revs, tree);\n+\t\t}\n \t\tctx->show_commit(commit, ctx->show_data);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\ndiff --git a/revision.c b/revision.c\nindex 062749437..6d355b43c 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -175,7 +175,6 @@ static void add_pending_object_with_path(struct rev_info *revs,\n \t\tstrbuf_release(&buf);\n \t\treturn; /* do not add the commit itself */\n \t}\n-\tobj->flags |= USER_GIVEN;\n \tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n }\n \ndiff --git a/revision.h b/revision.h\nindex c599c34da..cd6b62313 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -8,7 +8,11 @@\n #include \"diff.h\"\n #include \"commit-slab-decl.h\"\n \n-/* Remember to update object flag allocation in object.h */\n+/* Remember to update object flag allocation in object.h\n+ * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n+ * filtering trees and blobs, but it may be useful to support filtering commits\n+ * in the future.\n+ */\n #define SEEN\t\t(1u<<0)\n #define UNINTERESTING (1u<<1)\n #define TREESAME\t(1u<<2)\n@@ -20,9 +24,9 @@\n #define SYMMETRIC_LEFT\t(1u<<8)\n #define PATCHSAME\t(1u<<9)\n #define BOTTOM\t\t(1u<<10)\n-#define USER_GIVEN\t(1u<<25) /* given directly by the user */\n+#define NOT_USER_GIVEN\t(1u<<25) /* tree or blob not given directly by user */\n #define TRACK_LINEAR\t(1u<<26)\n-#define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n+#define ALL_REV_FLAGS\t(((1u<<11)-1) | NOT_USER_GIVEN | TRACK_LINEAR)\n \n #define DECORATE_SHORT_REFS\t1\n #define DECORATE_FULL_REFS\t2\n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355428","messageId":"92faf2e011474bcc87b1b0974e7e232f6469dc4a.1534183648.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534183648.git.matvore@google.com","subject":"[PATCH v3 5/5] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-13T18:14:32Z","receivedAt":"2018-08-13T18:15:08Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Teach list-objects the \"tree:0\" filter which allows for filtering\nout all tree and blob objects (unless other objects are explicitly\nspecified by the user). The purpose of this patch is to allow smaller\npartial clones.\n\nThe name of this filter - tree:0 - does not explicitly specify that\nit also filters out all blobs, but this should not cause much confusion\nbecause blobs are not at all useful without the trees that refer to\nthem.\n\nI also consider only:commits as a name, but this is inaccurate because\nit suggests that annotated tags are omitted, but actually they are\nincluded.\n\nThe name \"tree:0\" allows later filtering based on depth, i.e. \"tree:1\"\nwould filter out all but the root tree and blobs. In order to avoid\nconfusion between 0 and capital O, the documentation was worded in a\nsomewhat round-about way that also hints at this future improvement to\nthe feature.\n\nSigned-off-by: Matthew DeVore \n---\n Documentation/rev-list-options.txt | 3 ++\n list-objects-filter-options.c | 4 +++\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 50 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 27 ++++++++++++++\n t/t5616-partial-clone.sh | 27 ++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 13 +++++++\n 7 files changed, 125 insertions(+)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 7b273635d..9e351ec2a 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -743,6 +743,9 @@ specification contained in .\n \tA debug option to help with future \"partial clone\" development.\n \tThis option specifies how missing objects are handled.\n +\n+The form '--filter=tree:' omits all blobs and trees deeper than\n+ from the root tree. Currently, only =0 is supported.\n++\n The form '--missing=error' requests that rev-list stop with an error if\n a missing object is encountered. This is the default action.\n +\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex c0e2bd6a0..a28382940 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -50,6 +50,10 @@ static int gently_parse_list_objects_filter(\n \t\t\treturn 0;\n \t\t}\n \n+\t} else if (!strcmp(arg, \"tree:0\")) {\n+\t\tfilter_options->choice = LOFC_TREE_NONE;\n+\t\treturn 0;\n+\n \t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n \t\tstruct object_context oc;\n \t\tstruct object_id sparse_oid;\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 0000a61f8..af64e5c66 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -10,6 +10,7 @@ enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n+\tLOFC_TREE_NONE,\n \tLOFC_SPARSE_OID,\n \tLOFC_SPARSE_PATH,\n \tLOFC__COUNT /* must be last */\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex a0ba78b20..8e3caf5bf 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -80,6 +80,55 @@ static void *filter_blobs_none__init(\n \treturn d;\n }\n \n+/*\n+ * A filter for list-objects to omit ALL trees and blobs from the traversal.\n+ * Can OPTIONALLY collect a list of the omitted OIDs.\n+ */\n+struct filter_trees_none_data {\n+\tstruct oidset *omits;\n+};\n+\n+static enum list_objects_filter_result filter_trees_none(\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_trees_none_data *filter_data = filter_data_;\n+\n+\tswitch (filter_situation) {\n+\tdefault:\n+\t\tdie(\"unknown filter_situation\");\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFS_BEGIN_TREE:\n+\tcase LOFS_BLOB:\n+\t\tif (filter_data->omits)\n+\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n+\n+\tcase LOFS_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\treturn LOFR_ZERO;\n+\n+\t}\n+}\n+\n+static void* filter_trees_none__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn)\n+{\n+\tstruct filter_trees_none_data *d = xcalloc(1, sizeof(*d));\n+\td->omits = omitted;\n+\n+\t*filter_fn = filter_trees_none;\n+\t*filter_free_fn = free;\n+\treturn d;\n+}\n+\n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n@@ -374,6 +423,7 @@ static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n+\tfilter_trees_none__init,\n \tfilter_sparse_oid__init,\n \tfilter_sparse_path__init,\n };\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 5e35f33bf..65f2cf446 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -72,6 +72,33 @@ test_expect_success 'get an error for missing tree object' '\n \tgrep -q \"bad tree object\" bad_tree\n '\n \n+test_expect_success 'setup for tests of tree:0' '\n+\tmkdir r1/subtree &&\n+\techo \"This is a file in a subtree\" > r1/subtree/file &&\n+\tgit -C r1 add subtree/file &&\n+\tgit -C r1 commit -m subtree\n+'\n+\n+test_expect_success 'verify tree:0 packfile has no blobs or trees' '\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\t! grep -E \"tree|blob\" objs\n+'\n+\n+test_expect_success 'grab tree directly when using tree:0' '\n+\t# We should get the tree specified directly but not its blobs or subtrees.\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD:\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\tgrep -E \"tree|blob\" objs >trees_and_blobs &&\n+\ttest_line_count = 1 trees_and_blobs\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex bbbe7537d..fc4d182c0 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -170,6 +170,33 @@ test_expect_success 'partial clone fetches blobs pointed to by refs even if norm\n \tgit -C dst fsck\n '\n \n+test_expect_success 'can use tree:0 to filter partial clone' '\n+\trm -rf dst &&\n+\tgit clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n+\tgit -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n+\tcat fetched_objects \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n+\tsort fetched_types -u >unique_types.observed &&\n+\techo commit > unique_types.expected &&\n+\ttest_cmp unique_types.observed unique_types.expected\n+'\n+\n+test_expect_success 'show missing tree objects with --missing=print' '\n+\tgit -C dst rev-list master --missing=print --quiet --objects >missing_objs &&\n+\tsed \"s/?//\" missing_objs \\\n+\t\t| xargs -n1 git -C srv.bare cat-file -t \\\n+\t\t>missing_types &&\n+\tsort -u missing_types >missing_types.uniq &&\n+\techo tree >expected &&\n+\ttest_cmp missing_types.uniq expected\n+'\n+\n+test_expect_success 'do not complain when a missing tree cannot be parsed' '\n+\tgit -C dst rev-list master --missing=print --quiet --objects 2>rev_list_err >&2 &&\n+\t! grep -q \"Could not read \" rev_list_err\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 0a37dd5f9..6ccffddbc 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -196,6 +196,19 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n \ttest_cmp observed expected\n '\n \n+# Test tree:0 filter.\n+\n+test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n+\tgit -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=tree:0 \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed s/~// \\\n+\t\t| xargs -n1 git -C r3 cat-file -t \\\n+\t\t| sort -u >filtered_types &&\n+\tprintf \"blob\\ntree\\n\" > expected &&\n+\ttest_cmp filtered_types expected\n+'\n+\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.18.0.597.ga71716f1ad-goog\n\n"},{"id":"355431","messageId":"20180813182025.14229-1-jonathantanmy@google.com","threadId":"49086","inReplyTo":"6d6b06e870b5d6dbe9bb88bd80fcce34ba4dcccf.1533941986.git.matvore@google.com","subject":"Re: [PATCH v2 3/5] rev-list: handle missing tree objects properly","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-08-13T18:20:25Z","receivedAt":"2018-08-13T18:20:32Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> In list-objects.c we no longer print a message to stderr if a tree\n> object is missing (quiet_on_missing is always true). I couldn't find\n> any place where this would matter, or where the caller of\n> traverse_commit_list would need to be fixed to show the error. However,\n> in the future it would be trivial to make the caller show the message if\n> we needed to.\n\nIndeed, and I'm not sure why the message was there in the first place -\nif parsing fails when revs->ignore_missing_links and\nrevs->exclude_promisor_objects are both false, we print the OID anyway\nin the \"die\" call, so any message printed by parse_tree_gently() seems\nsuperfluous.\n\nIt might be better to add an additional commit that removes the \"gently\"\ncondition (in other words, always parsing gently), with a message\nexplaining the above. Also, in that commit, I prefer not to add the\n\"/*quiet_on_missing*/\" explanation (we don't seem to do that in Git\ncode); I also know that the \">= 0\" is a holdover from the existing \"< 0\"\ncode, but we don't need to do that either.\n\n> This is not tested very thoroughly, since we cannot create promisor\n> objects in tests without using an actual partial clone. t0410 has a\n> promise_and_delete utility function, but the is_promisor_object function\n> does not return 1 for objects deleted in this way. More tests will will\n> come in a patch that implements a filter that can be used with git\n> clone.\n\nis_promisor_object() should. If you still have the code you used to\nverify that, can you share it? In particular, pay attention to the path\nof the repo - promise_and_delete is hardcoded to use one particular\npath.\n\nWhether you test in this patch or in the last patch, make sure that the\nfollowing are tested:\n git rev-list --missing=error, allow-any, allow-promisor, print\n git rev-list --exclude-promisor-objects\n\nAlso, test when a tree pointed to by a commit is missing, and when a\ntree pointed to by a tree is missing.\n\n> @@ -152,20 +151,21 @@ static void process_tree(struct traversal_context *ctx,\n> \t\tdie(\"bad tree object\");\n> \tif (obj->flags & (UNINTERESTING | SEEN))\n> \t\treturn;\n> -\tif (parse_tree_gently(tree, gently) < 0) {\n> +\tparsed = parse_tree_gently(tree, /*quiet_on_missing=*/1) >= 0;\n> +\tif (!parsed) {\n> \t\tif (revs->ignore_missing_links)\n> \t\t\treturn;\n> \n> +\t\tif (!is_promisor_object(&obj->oid))\n> +\t\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n> +\n> \t\t/*\n> \t\t * Pre-filter known-missing tree objects when explicitly\n> \t\t * requested. This may cause the actual filter to report\n> \t\t * an incomplete list of missing objects.\n> \t\t */\n> -\t\tif (revs->exclude_promisor_objects &&\n> -\t\t is_promisor_object(&obj->oid))\n> +\t\tif (revs->exclude_promisor_objects)\n> \t\t\treturn;\n> -\n> -\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n> \t}\n\nThe missing mechanism (for error, allow-any, print) should work without\nneeding to consult whether an object is a promisor object or not - it\nshould just print whatever is missing, so the \"if\n(!is_promisor_object...\" line looks out of place.\n\nIn my original review [1], I suggested that we always show a tree if we\nhave its hash - if we don't have the object, we just recurse into it.\nThis would be the same as your patch, except that the 'die(\"bad tree\nobject...' is totally removed instead of merely moved. I still think\nthis solution has some merit - all the tests still pass (except that we\nneed to check for \"unable to read\" instead of \"bad tree object\" in error\nmessages), but I just realized that it might still be backwards\nincompatible in that a basic \"rev-list --objects\" would now succeed\ninstead of fail if a tree was missing (I haven't tested this though).\n\nWe might need a flag called \"do_not_die_on_missing_tree\" (much like your\noriginal idea of \"show_missing_trees\") so that callers that are prepared\nto deal with missing trees can set this. Sorry for the churn. You can\ndocument it as such:\n\n Blobs are shown without regard for their existence. But not so for\n trees: unless exclude_promisor_objects is set and the tree in question\n is a promisor object, or ignore_missing_links is set (and in this case,\n the tree in question may or may not be a promisor object), the revision\n walker dies with a \"bad tree object\" message when encountering a\n missing tree.\n\n For callers that can handle missing trees and want them to be\n filterable and showable, set this to true. The revision walker will\n filter and show such a missing tree as usual, but will not attempt to\n recurse into this tree object.\n\n[1] https://public-inbox.org/git/20180810002411.13447-1-jonathantanmy@google.com/\n"},{"id":"355435","messageId":"20180813182950.15723-1-jonathantanmy@google.com","threadId":"49086","inReplyTo":"0af4ef64aa52d76a597b4e97c0626c70fb2a883b.1533941986.git.matvore@google.com","subject":"Re: [PATCH v2 5/5] list-objects-filter: implement filter tree:none","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-08-13T18:29:50Z","receivedAt":"2018-08-13T18:29:58Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> -\tcase LOFS_BEGIN_TREE:\n> -\t\tassert(obj->type == OBJ_TREE);\n> -\t\t/* always include all tree objects */\n> -\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> -\n> \tcase LOFS_END_TREE:\n> \t\tassert(obj->type == OBJ_TREE);\n> \t\treturn LOFR_ZERO;\n> \n> +\tcase LOFS_BEGIN_TREE:\n> +\t\tassert(obj->type == OBJ_TREE);\n> +\t\tif (!filter_data->omit_trees)\n> +\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> +\n> +\t\t/*\n> +\t\t * Fallthrough to insert into omitted list for trees as well as\n> +\t\t * blobs.\n> +\t\t */\n> +\t\t/* fallthrough */\n> \tcase LOFS_BLOB:\n> -\t\tassert(obj->type == OBJ_BLOB);\n> \t\tassert((obj->flags & SEEN) == 0);\n\nAfter looking at the resulting file, I don't think saving a few lines of\ncode (to add the OID, then return LOFR_MARK_SEEN) is worth rearranging\nthe cases and falling through. Can you just add the OID-adding code to\nthe LOFS_BEGIN_TREE case?\n\n> +test_expect_success 'can use tree:none to filter partial clone' '\n> +\trm -rf dst &&\n> +\tgit clone --no-checkout --filter=tree:none \"file://$(pwd)/srv.bare\" dst &&\n> +\tgit -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n> +\tcat fetched_objects \\\n> +\t\t| awk -f print_1.awk \\\n> +\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n> +\tsort fetched_types -u >unique_types.observed &&\n> +\techo commit > unique_types.expected &&\n> +\ttest_cmp unique_types.observed unique_types.expected\n> +'\n\nWe also need to verify that the resulting partial clone works - after\nall relevant tests, can you also ensure that:\n - fsck works\n - a cat-file on an indirectly missing tree works (i.e. if you have\n commit -> A -> B and both A and B are missing, cat-file the B)\n - fsck still works after the cat-file\n\nThere is another potential issue about expanding the documentation of\nthe pack protocol because we now support a new type of filter, but that\nis fine because the protocol currently points us to the rev-list\ndocumentation (which is updated). We probably need a way for clients to\nquery servers about which filters they support, but that is definitely\nbeyond the scope of this patch set.\n\n> +test_expect_success 'show missing tree objects with --missing=print' '\n> +\tgit -C dst rev-list master --missing=print --quiet --objects >missing_objs &&\n> +\tsed \"s/?//\" missing_objs \\\n> +\t\t| xargs -n1 git -C srv.bare cat-file -t \\\n> +\t\t>missing_types &&\n> +\tsort -u missing_types >missing_types.uniq &&\n> +\techo tree >expected &&\n> +\ttest_cmp missing_types.uniq expected\n> +'\n\nAs stated in my review of patch 3, also test the other --missing\narguments.\n\nPatches 1, 2, and 4 look good to me. (Writing this here so that I don't\nneed to send one e-mail for each.)\n"},{"id":"355506","messageId":"CAMfpvh+awO8kpVtxKxMDuiXx2XM0o3xGycXdSF+ZJ0NXyzv5Hg@mail.gmail.com","threadId":"49086","inReplyTo":"20180813182025.14229-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 3/5] rev-list: handle missing tree objects properly","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-14T00:22:38Z","receivedAt":"2018-08-14T00:22:53Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Resending this in plain-text mode so that git@vger.kernel.org won't\nbounce it. Sorry for those of you receiving this twice.\n\nOn Mon, Aug 13, 2018 at 11:20 AM Jonathan Tan wrote:\n>\n> > In list-objects.c we no longer print a message to stderr if a tree\n> > object is missing (quiet_on_missing is always true). I couldn't find\n> > any place where this would matter, or where the caller of\n> > traverse_commit_list would need to be fixed to show the error. However,\n> > in the future it would be trivial to make the caller show the message if\n> > we needed to.\n>\n> Indeed, and I'm not sure why the message was there in the first place -\n> if parsing fails when revs->ignore_missing_links and\n> revs->exclude_promisor_objects are both false, we print the OID anyway\n> in the \"die\" call, so any message printed by parse_tree_gently() seems\n> superfluous.\n>\n> It might be better to add an additional commit that removes the \"gently\"\n> condition (in other words, always parsing gently), with a message\n> explaining the above. Also, in that commit, I prefer not to add the\n> \"/*quiet_on_missing*/\" explanation (we don't seem to do that in Git\n> code); I also know that the \">= 0\" is a holdover from the existing \"< 0\"\n> code, but we don't need to do that either.\nGood idea. I've added a new commit which replaces the calculation with\na hard-coded \"1\"\nI don't understand about the \">= 0\". What should I replace it with?\nMaybe you mean the return is never positive so I can change:\n\nparse_tree_gently(tree, 1) >= 0\n\nto:\n!parse_tree_gently(tree, 1)\n\n?\n\n>\n> > This is not tested very thoroughly, since we cannot create promisor\n> > objects in tests without using an actual partial clone. t0410 has a\n> > promise_and_delete utility function, but the is_promisor_object function\n> > does not return 1 for objects deleted in this way. More tests will will\n> > come in a patch that implements a filter that can be used with git\n> > clone.\n>\n> is_promisor_object() should. If you still have the code you used to\n> verify that, can you share it? In particular, pay attention to the path\n> of the repo - promise_and_delete is hardcoded to use one particular\n> path.\nIt turns out I wasn't setting the extensions.partial_clone config in\nmy test, and that's why everything wasn't working. So I've moved all\nthe tests feasible back to the earlier commit. Cool :)\n\n>\n> Whether you test in this patch or in the last patch, make sure that the\n> following are tested:\n> git rev-list --missing=error, allow-any, allow-promisor, print\n> git rev-list --exclude-promisor-objects\n>\nAdded --missing=print, --missing=allow-any, and\n--exclude-promisor-objects to t0410\n--missing=allow-promisor did some seem sufficiently interesting or\ndifferent from allow-any to justify adding it.\nI had to put missing=error into the commit that introduces the tree:0\nfilter, since that flag causes an automatic attempt to fetch the\nmissing object, which t0410 does not seem to support. So added test\ncase \"auto-fetching of trees with --missing=error\" to t5616.\n\n> Also, test when a tree pointed to by a commit is missing, and when a\n> tree pointed to by a tree is missing.\nFormer is done multiple times already, added latter to t0410 as\n\"missing non-root tree object and rev-list.\"\n>\n> > @@ -152,20 +151,21 @@ static void process_tree(struct traversal_context *ctx,\n> > die(\"bad tree object\");\n> > if (obj->flags & (UNINTERESTING | SEEN))\n> > return;\n> > - if (parse_tree_gently(tree, gently) < 0) {\n> > + parsed = parse_tree_gently(tree, /*quiet_on_missing=*/1) >= 0;\n> > + if (!parsed) {\n> > if (revs->ignore_missing_links)\n> > return;\n> >\n> > + if (!is_promisor_object(&obj->oid))\n> > + die(\"bad tree object %s\", oid_to_hex(&obj->oid));\n> > +\n> > /*\n> > * Pre-filter known-missing tree objects when explicitly\n> > * requested. This may cause the actual filter to report\n> > * an incomplete list of missing objects.\n> > */\n> > - if (revs->exclude_promisor_objects &&\n> > - is_promisor_object(&obj->oid))\n> > + if (revs->exclude_promisor_objects)\n> > return;\n> > -\n> > - die(\"bad tree object %s\", oid_to_hex(&obj->oid));\n> > }\n>\n> The missing mechanism (for error, allow-any, print) should work without\n> needing to consult whether an object is a promisor object or not - it\n> should just print whatever is missing, so the \"if\n> (!is_promisor_object...\" line looks out of place.\nDone. I considered that a missing object which is not a promisor is a\nserious error, so I had it die here. But now that I've added the\ndo_not_die_on_missing_tree flag, it's more natural to keep the\nprevious promisor check as-is. Also, is_promisor_object is an\nexpensive check, and it would be better to skip it during the common\nexecution path (which should be when exclude_promisor_objects, an\ninternal-use-only flag, is *not* set, which means we never call\nis_promisor_object.\n\n>\n> In my original review [1], I suggested that we always show a tree if we\n> have its hash - if we don't have the object, we just recurse into it.\n> This would be the same as your patch, except that the 'die(\"bad tree\n> object...' is totally removed instead of merely moved. I still think\n> this solution has some merit - all the tests still pass (except that we\n> need to check for \"unable to read\" instead of \"bad tree object\" in error\n> messages), but I just realized that it might still be backwards\n> incompatible in that a basic \"rev-list --objects\" would now succeed\n> instead of fail if a tree was missing (I haven't tested this though).\nThe presence of the die if !is_promisor_object is what justified the\nchanging of the parse_tree_gently to always be gently, since it is\nwhat showed the OID. Can we really remove both? Maybe in a different\npatch set, since I'm no longer touching that line?\n\n>\n> We might need a flag called \"do_not_die_on_missing_tree\" (much like your\n> original idea of \"show_missing_trees\") so that callers that are prepared\n> to deal with missing trees can set this. Sorry for the churn. You can\n> document it as such:\nAdded it, but not with a command-line flag, only in rev-info.h. We can\nalways add a flag later if people have been relying on the existing\nbehavior of git rev-list to balk at missing trees. (That seems\nunlikely though, considering there is no filter to enable that before\nthis patchset).\n\n>\n> Blobs are shown without regard for their existence. But not so for\n> trees: unless exclude_promisor_objects is set and the tree in question\n> is a promisor object, or ignore_missing_links is set (and in this case,\n> the tree in question may or may not be a promisor object), the revision\n> walker dies with a \"bad tree object\" message when encountering a\n> missing tree.\n>\n> For callers that can handle missing trees and want them to be\n> filterable and showable, set this to true. The revision walker will\n> filter and show such a missing tree as usual, but will not attempt to\n> recurse into this tree object.\n>\n> [1] https://public-inbox.org/git/20180810002411.13447-1-jonathantanmy@google.com/\n"},{"id":"355508","messageId":"CAMfpvhLhC+e6JhA+H6WRh1kcPwCORrLhRfqoExTWh3Z36wJNjQ@mail.gmail.com","threadId":"49086","inReplyTo":"20180813182950.15723-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 5/5] list-objects-filter: implement filter tree:none","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-14T00:55:01Z","receivedAt":"2018-08-14T00:55:54Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Mon, Aug 13, 2018 at 11:29 AM Jonathan Tan wrote:\n>\n> > - case LOFS_BEGIN_TREE:\n> > - assert(obj->type == OBJ_TREE);\n> > - /* always include all tree objects */\n> > - return LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> > -\n> > case LOFS_END_TREE:\n> > assert(obj->type == OBJ_TREE);\n> > return LOFR_ZERO;\n> >\n> > + case LOFS_BEGIN_TREE:\n> > + assert(obj->type == OBJ_TREE);\n> > + if (!filter_data->omit_trees)\n> > + return LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> > +\n> > + /*\n> > + * Fallthrough to insert into omitted list for trees as well as\n> > + * blobs.\n> > + */\n> > + /* fallthrough */\n> > case LOFS_BLOB:\n> > - assert(obj->type == OBJ_BLOB);\n> > assert((obj->flags & SEEN) == 0);\n>\n> After looking at the resulting file, I don't think saving a few lines of\n> code (to add the OID, then return LOFR_MARK_SEEN) is worth rearranging\n> the cases and falling through. Can you just add the OID-adding code to\n> the LOFS_BEGIN_TREE case?\n\nI've followed Jeff's suggestion of splitting up the functions, so\nthough the code is now redundant, it is ready for some future\nimprovements that are planned, and it's more clear and consistent with\nthe other filter functions.\n\n>\n> > +test_expect_success 'can use tree:none to filter partial clone' '\n> > + rm -rf dst &&\n> > + git clone --no-checkout --filter=tree:none \"file://$(pwd)/srv.bare\" dst &&\n> > + git -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n> > + cat fetched_objects \\\n> > + | awk -f print_1.awk \\\n> > + | xargs -n1 git -C dst cat-file -t >fetched_types &&\n> > + sort fetched_types -u >unique_types.observed &&\n> > + echo commit > unique_types.expected &&\n> > + test_cmp unique_types.observed unique_types.expected\n> > +'\n>\n> We also need to verify that the resulting partial clone works - after\n> all relevant tests, can you also ensure that:\n> - fsck works\n> - a cat-file on an indirectly missing tree works (i.e. if you have\n> commit -> A -> B and both A and B are missing, cat-file the B)\n> - fsck still works after the cat-file\nDone - added it to t5616 in the last commit of the patchset.\n\n>\n> There is another potential issue about expanding the documentation of\n> the pack protocol because we now support a new type of filter, but that\n> is fine because the protocol currently points us to the rev-list\n> documentation (which is updated). We probably need a way for clients to\n> query servers about which filters they support, but that is definitely\n> beyond the scope of this patch set.\n>\n> > +test_expect_success 'show missing tree objects with --missing=print' '\n> > + git -C dst rev-list master --missing=print --quiet --objects >missing_objs &&\n> > + sed \"s/?//\" missing_objs \\\n> > + | xargs -n1 git -C srv.bare cat-file -t \\\n> > + >missing_types &&\n> > + sort -u missing_types >missing_types.uniq &&\n> > + echo tree >expected &&\n> > + test_cmp missing_types.uniq expected\n> > +'\n>\n> As stated in my review of patch 3, also test the other --missing\n> arguments.\nDone, mostly in t0410 in patch 3.\n\n>\n> Patches 1, 2, and 4 look good to me. (Writing this here so that I don't\n> need to send one e-mail for each.)\n"},{"id":"355509","messageId":"CAMfpvhJMhNY4+raVN_FBOwkk48=RsKL40Mtk74eH2o9+8m4+=Q@mail.gmail.com","threadId":"49086","inReplyTo":"ce743b3b-9d60-a6d7-9064-e9272c875e00@jeffhostetler.com","subject":"Re: [PATCH v2 5/5] list-objects-filter: implement filter tree:none","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-14T00:57:01Z","receivedAt":"2018-08-14T00:57:16Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Mon, Aug 13, 2018 at 9:38 AM Jeff Hostetler wrote:\n>\n>\n>\n> On 8/10/2018 7:06 PM, Matthew DeVore wrote:\n> > Teach list-objects the \"tree:none\" filter which allows for filtering\n> > out all tree and blob objects (unless other objects are explicitly\n> > specified by the user). The purpose of this patch is to allow smaller\n> > partial clones.\n> >\n> > The name of this filter - tree:none - does not explicitly specify that\n> > it also filters out all blobs, but this should not cause much confusion\n> > because blobs are not at all useful without the trees that refer to\n> > them.\n> >\n> > I also consider only:commits as a name, but this is inaccurate because\n> > it suggests that annotated tags are omitted, but actually they are\n> > included.\n> >\n> > Signed-off-by: Matthew DeVore \n> > ---\n> > Documentation/rev-list-options.txt | 2 ++\n> > list-objects-filter-options.c | 4 +++\n> > list-objects-filter-options.h | 1 +\n> > list-objects-filter.c | 49 +++++++++++++++++++-------\n> > t/t5317-pack-objects-filter-objects.sh | 27 ++++++++++++++\n> > t/t5616-partial-clone.sh | 27 ++++++++++++++\n> > t/t6112-rev-list-filters-objects.sh | 13 +++++++\n> > 7 files changed, 110 insertions(+), 13 deletions(-)\n> >\n> > diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\n> > index 7b273635d..68b4b9552 100644\n> > --- a/Documentation/rev-list-options.txt\n> > +++ b/Documentation/rev-list-options.txt\n> > @@ -743,6 +743,8 @@ specification contained in .\n> > A debug option to help with future \"partial clone\" development.\n> > This option specifies how missing objects are handled.\n> > +\n> > +The form '--filter=tree:none' omits all blobs and trees.\n> > ++\n> > The form '--missing=error' requests that rev-list stop with an error if\n> > a missing object is encountered. This is the default action.\n> > +\n> > diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\n> > index c0e2bd6a0..523cb00a0 100644\n> > --- a/list-objects-filter-options.c\n> > +++ b/list-objects-filter-options.c\n> > @@ -50,6 +50,10 @@ static int gently_parse_list_objects_filter(\n> > return 0;\n> > }\n> >\n> > + } else if (!strcmp(arg, \"tree:none\")) {\n> > + filter_options->choice = LOFC_TREE_NONE;\n> > + return 0;\n> > +\n> > } else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n> > struct object_context oc;\n> > struct object_id sparse_oid;\n> > diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\n> > index 0000a61f8..af64e5c66 100644\n> > --- a/list-objects-filter-options.h\n> > +++ b/list-objects-filter-options.h\n> > @@ -10,6 +10,7 @@ enum list_objects_filter_choice {\n> > LOFC_DISABLED = 0,\n> > LOFC_BLOB_NONE,\n> > LOFC_BLOB_LIMIT,\n> > + LOFC_TREE_NONE,\n> > LOFC_SPARSE_OID,\n> > LOFC_SPARSE_PATH,\n> > LOFC__COUNT /* must be last */\n> > diff --git a/list-objects-filter.c b/list-objects-filter.c\n> > index a0ba78b20..22c894093 100644\n> > --- a/list-objects-filter.c\n> > +++ b/list-objects-filter.c\n> > @@ -26,38 +26,45 @@\n> > #define FILTER_SHOWN_BUT_REVISIT (1<<21)\n> >\n> > /*\n> > - * A filter for list-objects to omit ALL blobs from the traversal.\n> > - * And to OPTIONALLY collect a list of the omitted OIDs.\n> > + * A filter for list-objects to omit ALL blobs from the traversal, and possibly\n> > + * trees as well.\n> > + * Can OPTIONALLY collect a list of the omitted OIDs.\n> > */\n> > -struct filter_blobs_none_data {\n> > +struct filter_none_of_type_data {\n> > + /* blobs are always omitted */\n> > + unsigned omit_trees : 1;\n> > struct oidset *omits;\n> > };\n> >\n>\n> I'm not sure I'd convert the existing filter types.\n> When I created this file, I created a set of function pairs\n> for each filter type:\n> filter_() and filter___init()\n>\n> with the latter being added to the s_filters[] array and created\n> a choice enum having corresponding values\n> LOFC_\n>\n> Here you're adding a new _init() and LOFC_ key, but mapping both\n> the original \"blob:none\" and the new \"tree:none\" to a combined\n> filter function and blends these 2 modes.\n>\n> Style-wise, I'd keep the original filters as they were and add a\n> new function pair for the new tree:none filter. Then you can\n> simplify the logic inside your new filter. For example, in your\n> filter \"filter_data->omit_trees\" will always be true, so you can\n> just do the \"if (filter_data->omits) oidset_insert(...); return _SEEN\"\n> and not have the fallthru stuff -- or get rid of the asserts() and put\n> the case labels together.\n>\n> One of the things I wanted to do (when I found some free time) was to\n> add a \"tree:none\" and maybe a \"tree:root\" filter. (The latter only\n> including the root trees associated with the fetched commits, since\n> there are/were some places where we implicitly also load the root tree\n> when loading the commit object.) So in that vein, it might be that we\n> would want a \"tree:\" filter instead with 0 = none and 1 = root.\n> I wasn't ready to propose that when I did the filtering, but I had that\n> in mind. (And is partially why I suggest keeping your new filter\n> independent of the existing ones.)\n\nThat's fair. I've split up the functions to be completely separate,\nand changed the filter name to tree:0 so it can later be extended as\nyou suggest.\n\n>\n> Jeff\n>\n>\n> > -static enum list_objects_filter_result filter_blobs_none(\n> > +static enum list_objects_filter_result filter_none_of_type(\n> > enum list_objects_filter_situation filter_situation,\n> > struct object *obj,\n> > const char *pathname,\n> > const char *filename,\n> > void *filter_data_)\n> > {\n> > - struct filter_blobs_none_data *filter_data = filter_data_;\n> > + struct filter_none_of_type_data *filter_data = filter_data_;\n> >\n> > switch (filter_situation) {\n> > default:\n> > die(\"unknown filter_situation\");\n> > return LOFR_ZERO;\n> >\n> > - case LOFS_BEGIN_TREE:\n> > - assert(obj->type == OBJ_TREE);\n> > - /* always include all tree objects */\n> > - return LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> > -\n> > case LOFS_END_TREE:\n> > assert(obj->type == OBJ_TREE);\n> > return LOFR_ZERO;\n> >\n> > + case LOFS_BEGIN_TREE:\n> > + assert(obj->type == OBJ_TREE);\n> > + if (!filter_data->omit_trees)\n> > + return LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> > +\n> > + /*\n> > + * Fallthrough to insert into omitted list for trees as well as\n> > + * blobs.\n> > + */\n> > + /* fallthrough */\n> > case LOFS_BLOB:\n> > - assert(obj->type == OBJ_BLOB);\n> > assert((obj->flags & SEEN) == 0);\n> >\n> > if (filter_data->omits)\n> > @@ -72,10 +79,25 @@ static void *filter_blobs_none__init(\n> > filter_object_fn *filter_fn,\n> > filter_free_fn *filter_free_fn)\n> > {\n> > - struct filter_blobs_none_data *d = xcalloc(1, sizeof(*d));\n> > + struct filter_none_of_type_data *d = xcalloc(1, sizeof(*d));\n> > + d->omits = omitted;\n> > +\n> > + *filter_fn = filter_none_of_type;\n> > + *filter_free_fn = free;\n> > + return d;\n> > +}\n> > +\n> > +static void* filter_tree_none__init(\n> > + struct oidset *omitted,\n> > + struct list_objects_filter_options *filter_options,\n> > + filter_object_fn *filter_fn,\n> > + filter_free_fn *filter_free_fn)\n> > +{\n> > + struct filter_none_of_type_data *d = xcalloc(1, sizeof(*d));\n> > + d->omit_trees = 1;\n> > d->omits = omitted;\n> >\n> > - *filter_fn = filter_blobs_none;\n> > + *filter_fn = filter_none_of_type;\n> > *filter_free_fn = free;\n> > return d;\n> > }\n> > @@ -374,6 +396,7 @@ static filter_init_fn s_filters[] = {\n> > NULL,\n> > filter_blobs_none__init,\n> > filter_blobs_limit__init,\n> > + filter_tree_none__init,\n> > filter_sparse_oid__init,\n> > filter_sparse_path__init,\n> > };\n> > diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\n> > index 5e35f33bf..28a8c916a 100755\n> > --- a/t/t5317-pack-objects-filter-objects.sh\n> > +++ b/t/t5317-pack-objects-filter-objects.sh\n> > @@ -72,6 +72,33 @@ test_expect_success 'get an error for missing tree object' '\n> > grep -q \"bad tree object\" bad_tree\n> > '\n> >\n> > +test_expect_success 'setup for tests of tree:none' '\n> > + mkdir r1/subtree &&\n> > + echo \"This is a file in a subtree\" > r1/subtree/file &&\n> > + git -C r1 add subtree/file &&\n> > + git -C r1 commit -m subtree\n> > +'\n> > +\n> > +test_expect_success 'verify tree:none packfile has no blobs or trees' '\n> > + git -C r1 pack-objects --rev --stdout --filter=tree:none >commitsonly.pack <<-EOF &&\n> > + HEAD\n> > + EOF\n> > + git -C r1 index-pack ../commitsonly.pack &&\n> > + git -C r1 verify-pack -v ../commitsonly.pack >objs &&\n> > + ! grep -E \"tree|blob\" objs\n> > +'\n> > +\n> > +test_expect_success 'grab tree directly when using tree:none' '\n> > + # We should get the tree specified directly but not its blobs or subtrees.\n> > + git -C r1 pack-objects --rev --stdout --filter=tree:none >commitsonly.pack <<-EOF &&\n> > + HEAD:\n> > + EOF\n> > + git -C r1 index-pack ../commitsonly.pack &&\n> > + git -C r1 verify-pack -v ../commitsonly.pack >objs &&\n> > + grep -E \"tree|blob\" objs >trees_and_blobs &&\n> > + test_line_count = 1 trees_and_blobs\n> > +'\n> > +\n> > # Test blob:limit=[kmg] filter.\n> > # We boundary test around the size parameter. The filter is strictly less than\n> > # the value, so size 500 and 1000 should have the same results, but 1001 should\n> > diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\n> > index bbbe7537d..4fc068716 100755\n> > --- a/t/t5616-partial-clone.sh\n> > +++ b/t/t5616-partial-clone.sh\n> > @@ -170,6 +170,33 @@ test_expect_success 'partial clone fetches blobs pointed to by refs even if norm\n> > git -C dst fsck\n> > '\n> >\n> > +test_expect_success 'can use tree:none to filter partial clone' '\n> > + rm -rf dst &&\n> > + git clone --no-checkout --filter=tree:none \"file://$(pwd)/srv.bare\" dst &&\n> > + git -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n> > + cat fetched_objects \\\n> > + | awk -f print_1.awk \\\n> > + | xargs -n1 git -C dst cat-file -t >fetched_types &&\n> > + sort fetched_types -u >unique_types.observed &&\n> > + echo commit > unique_types.expected &&\n> > + test_cmp unique_types.observed unique_types.expected\n> > +'\n> > +\n> > +test_expect_success 'show missing tree objects with --missing=print' '\n> > + git -C dst rev-list master --missing=print --quiet --objects >missing_objs &&\n> > + sed \"s/?//\" missing_objs \\\n> > + | xargs -n1 git -C srv.bare cat-file -t \\\n> > + >missing_types &&\n> > + sort -u missing_types >missing_types.uniq &&\n> > + echo tree >expected &&\n> > + test_cmp missing_types.uniq expected\n> > +'\n> > +\n> > +test_expect_success 'do not complain when a missing tree cannot be parsed' '\n> > + git -C dst rev-list master --missing=print --quiet --objects 2>rev_list_err >&2 &&\n> > + ! grep -q \"Could not read \" rev_list_err\n> > +'\n> > +\n> > . \"$TEST_DIRECTORY\"/lib-httpd.sh\n> > start_httpd\n> >\n> > diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\n> > index 0a37dd5f9..ecdf6b4c3 100755\n> > --- a/t/t6112-rev-list-filters-objects.sh\n> > +++ b/t/t6112-rev-list-filters-objects.sh\n> > @@ -196,6 +196,19 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n> > test_cmp observed expected\n> > '\n> >\n> > +# Test tree:none filter.\n> > +\n> > +test_expect_success 'verify tree:none includes trees in \"filtered\" output' '\n> > + git -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=tree:none \\\n> > + | awk -f print_1.awk \\\n> > + | sed s/~// \\\n> > + | xargs -n1 git -C r3 cat-file -t \\\n> > + | sort -u >filtered_types &&\n> > + printf \"blob\\ntree\\n\" > expected &&\n> > + test_cmp filtered_types expected\n> > +'\n> > +\n> > +\n> > # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n> > # This models previously omitted objects that we did not receive.\n> >\n> >\n"},{"id":"355553","messageId":"d751d56b-84bb-a03d-5f2a-7dbaf8d947cc@jeffhostetler.com","threadId":"49086","inReplyTo":"92faf2e011474bcc87b1b0974e7e232f6469dc4a.1534183648.git.matvore@google.com","subject":"Re: [PATCH v3 5/5] list-objects-filter: implement filter tree:0","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2018-08-14T15:13:39Z","receivedAt":"2018-08-14T15:13:44Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 8/13/2018 2:14 PM, Matthew DeVore wrote:\n> Teach list-objects the \"tree:0\" filter which allows for filtering\n> out all tree and blob objects (unless other objects are explicitly\n> specified by the user). The purpose of this patch is to allow smaller\n> partial clones.\n> \n> The name of this filter - tree:0 - does not explicitly specify that\n> it also filters out all blobs, but this should not cause much confusion\n> because blobs are not at all useful without the trees that refer to\n> them.\n> \n> I also consider only:commits as a name, but this is inaccurate because\n> it suggests that annotated tags are omitted, but actually they are\n> included.\n> \n> The name \"tree:0\" allows later filtering based on depth, i.e. \"tree:1\"\n> would filter out all but the root tree and blobs. In order to avoid\n> confusion between 0 and capital O, the documentation was worded in a\n> somewhat round-about way that also hints at this future improvement to\n> the feature.\n> \n> Signed-off-by: Matthew DeVore \n> ---\n> Documentation/rev-list-options.txt | 3 ++\n> list-objects-filter-options.c | 4 +++\n> list-objects-filter-options.h | 1 +\n> list-objects-filter.c | 50 ++++++++++++++++++++++++++\n> t/t5317-pack-objects-filter-objects.sh | 27 ++++++++++++++\n> t/t5616-partial-clone.sh | 27 ++++++++++++++\n> t/t6112-rev-list-filters-objects.sh | 13 +++++++\n> 7 files changed, 125 insertions(+)\n> \n> diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\n> index 7b273635d..9e351ec2a 100644\n> --- a/Documentation/rev-list-options.txt\n> +++ b/Documentation/rev-list-options.txt\n> @@ -743,6 +743,9 @@ specification contained in .\n> \tA debug option to help with future \"partial clone\" development.\n> \tThis option specifies how missing objects are handled.\n> +\n> +The form '--filter=tree:' omits all blobs and trees deeper than\n> + from the root tree. Currently, only =0 is supported.\n> ++\n> The form '--missing=error' requests that rev-list stop with an error if\n> a missing object is encountered. This is the default action.\n> +\n> diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\n> index c0e2bd6a0..a28382940 100644\n> --- a/list-objects-filter-options.c\n> +++ b/list-objects-filter-options.c\n> @@ -50,6 +50,10 @@ static int gently_parse_list_objects_filter(\n> \t\t\treturn 0;\n> \t\t}\n> \n> +\t} else if (!strcmp(arg, \"tree:0\")) {\n> +\t\tfilter_options->choice = LOFC_TREE_NONE;\n> +\t\treturn 0;\n> +\n> \t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n> \t\tstruct object_context oc;\n> \t\tstruct object_id sparse_oid;\n> diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\n> index 0000a61f8..af64e5c66 100644\n> --- a/list-objects-filter-options.h\n> +++ b/list-objects-filter-options.h\n> @@ -10,6 +10,7 @@ enum list_objects_filter_choice {\n> \tLOFC_DISABLED = 0,\n> \tLOFC_BLOB_NONE,\n> \tLOFC_BLOB_LIMIT,\n> +\tLOFC_TREE_NONE,\n> \tLOFC_SPARSE_OID,\n> \tLOFC_SPARSE_PATH,\n> \tLOFC__COUNT /* must be last */\n> diff --git a/list-objects-filter.c b/list-objects-filter.c\n> index a0ba78b20..8e3caf5bf 100644\n> --- a/list-objects-filter.c\n> +++ b/list-objects-filter.c\n> @@ -80,6 +80,55 @@ static void *filter_blobs_none__init(\n> \treturn d;\n> }\n> \n> +/*\n> + * A filter for list-objects to omit ALL trees and blobs from the traversal.\n> + * Can OPTIONALLY collect a list of the omitted OIDs.\n> + */\n> +struct filter_trees_none_data {\n> +\tstruct oidset *omits;\n> +};\n> +\n> +static enum list_objects_filter_result filter_trees_none(\n> +\tenum list_objects_filter_situation filter_situation,\n> +\tstruct object *obj,\n> +\tconst char *pathname,\n> +\tconst char *filename,\n> +\tvoid *filter_data_)\n> +{\n> +\tstruct filter_trees_none_data *filter_data = filter_data_;\n> +\n> +\tswitch (filter_situation) {\n> +\tdefault:\n> +\t\tdie(\"unknown filter_situation\");\n> +\t\treturn LOFR_ZERO;\n> +\n> +\tcase LOFS_BEGIN_TREE:\n> +\tcase LOFS_BLOB:\n> +\t\tif (filter_data->omits)\n> +\t\t\toidset_insert(filter_data->omits, &obj->oid);\n> +\t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n> +\n> +\tcase LOFS_END_TREE:\n> +\t\tassert(obj->type == OBJ_TREE);\n> +\t\treturn LOFR_ZERO;\n> +\n> +\t}\n> +}\n\nThere are a couple of options here:\n[] If really want to omit all trees and blobs (and we DO NOT want\n the oidset of everything omitted), then we might be able to\n shortcut the traversal and speed things up.\n\n {} add a LOFR_SKIP_TREE bit to list_objects_filter_result\n {} test this bit process_tree() and avoid the init_tree_desc() and\n the while loop and some adjacent setup/tear-down code.\n {} make this filter something like:\n\n\tcase LOFS_BEGIN_TREE:\n\t\tif (filter_data->omits) {\n\t\t\toidset_insert(filter_data->omits, &obj->oid);\n\t\t\treturn LOFR_MARK_SEEN; /* ... (hard omit) */\n\t\t} else\n\t\t\treturn LOFR_SKIP_TREE;\n\tcase LOFS_BLOB:\n\t\tif (filter_data->omits) {\n\t\t\toidset_insert(filter_data->omits, &obj->oid);\n\t\t\treturn LOFR_MARK_SEEN; /* ... (hard omit) */\n\t\telse\n\t\t\tassert(...should not happen...);\n\n[] Later, if we choose to actually support a depth>0, we'll probably\n want a different filter function to conditionally include/exclude\n blobs, include shallow tree[node]s, and do some of the provisional-\n omit logic on deep tree[nodes] (in case a tree appears at multiple\n places/depths in the history). But that can wait.\n\nJeff\n\n\n> +\n> +static void* filter_trees_none__init(\n> +\tstruct oidset *omitted,\n> +\tstruct list_objects_filter_options *filter_options,\n> +\tfilter_object_fn *filter_fn,\n> +\tfilter_free_fn *filter_free_fn)\n> +{\n> +\tstruct filter_trees_none_data *d = xcalloc(1, sizeof(*d));\n> +\td->omits = omitted;\n> +\n> +\t*filter_fn = filter_trees_none;\n> +\t*filter_free_fn = free;\n> +\treturn d;\n> +}\n> +\n> /*\n> * A filter for list-objects to omit large blobs.\n> * And to OPTIONALLY collect a list of the omitted OIDs.\n> @@ -374,6 +423,7 @@ static filter_init_fn s_filters[] = {\n> \tNULL,\n> \tfilter_blobs_none__init,\n> \tfilter_blobs_limit__init,\n> +\tfilter_trees_none__init,\n> \tfilter_sparse_oid__init,\n> \tfilter_sparse_path__init,\n> };\n> diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\n> index 5e35f33bf..65f2cf446 100755\n> --- a/t/t5317-pack-objects-filter-objects.sh\n> +++ b/t/t5317-pack-objects-filter-objects.sh\n> @@ -72,6 +72,33 @@ test_expect_success 'get an error for missing tree object' '\n> \tgrep -q \"bad tree object\" bad_tree\n> '\n> \n> +test_expect_success 'setup for tests of tree:0' '\n> +\tmkdir r1/subtree &&\n> +\techo \"This is a file in a subtree\" > r1/subtree/file &&\n> +\tgit -C r1 add subtree/file &&\n> +\tgit -C r1 commit -m subtree\n> +'\n> +\n> +test_expect_success 'verify tree:0 packfile has no blobs or trees' '\n> +\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n> +\tHEAD\n> +\tEOF\n> +\tgit -C r1 index-pack ../commitsonly.pack &&\n> +\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n> +\t! grep -E \"tree|blob\" objs\n> +'\n> +\n> +test_expect_success 'grab tree directly when using tree:0' '\n> +\t# We should get the tree specified directly but not its blobs or subtrees.\n> +\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n> +\tHEAD:\n> +\tEOF\n> +\tgit -C r1 index-pack ../commitsonly.pack &&\n> +\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n> +\tgrep -E \"tree|blob\" objs >trees_and_blobs &&\n> +\ttest_line_count = 1 trees_and_blobs\n> +'\n> +\n> # Test blob:limit=[kmg] filter.\n> # We boundary test around the size parameter. The filter is strictly less than\n> # the value, so size 500 and 1000 should have the same results, but 1001 should\n> diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\n> index bbbe7537d..fc4d182c0 100755\n> --- a/t/t5616-partial-clone.sh\n> +++ b/t/t5616-partial-clone.sh\n> @@ -170,6 +170,33 @@ test_expect_success 'partial clone fetches blobs pointed to by refs even if norm\n> \tgit -C dst fsck\n> '\n> \n> +test_expect_success 'can use tree:0 to filter partial clone' '\n> +\trm -rf dst &&\n> +\tgit clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n> +\tgit -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n> +\tcat fetched_objects \\\n> +\t\t| awk -f print_1.awk \\\n> +\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n> +\tsort fetched_types -u >unique_types.observed &&\n> +\techo commit > unique_types.expected &&\n> +\ttest_cmp unique_types.observed unique_types.expected\n> +'\n> +\n> +test_expect_success 'show missing tree objects with --missing=print' '\n> +\tgit -C dst rev-list master --missing=print --quiet --objects >missing_objs &&\n> +\tsed \"s/?//\" missing_objs \\\n> +\t\t| xargs -n1 git -C srv.bare cat-file -t \\\n> +\t\t>missing_types &&\n> +\tsort -u missing_types >missing_types.uniq &&\n> +\techo tree >expected &&\n> +\ttest_cmp missing_types.uniq expected\n> +'\n> +\n> +test_expect_success 'do not complain when a missing tree cannot be parsed' '\n> +\tgit -C dst rev-list master --missing=print --quiet --objects 2>rev_list_err >&2 &&\n> +\t! grep -q \"Could not read \" rev_list_err\n> +'\n> +\n> . \"$TEST_DIRECTORY\"/lib-httpd.sh\n> start_httpd\n> \n> diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\n> index 0a37dd5f9..6ccffddbc 100755\n> --- a/t/t6112-rev-list-filters-objects.sh\n> +++ b/t/t6112-rev-list-filters-objects.sh\n> @@ -196,6 +196,19 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n> \ttest_cmp observed expected\n> '\n> \n> +# Test tree:0 filter.\n> +\n> +test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n> +\tgit -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=tree:0 \\\n> +\t\t| awk -f print_1.awk \\\n> +\t\t| sed s/~// \\\n> +\t\t| xargs -n1 git -C r3 cat-file -t \\\n> +\t\t| sort -u >filtered_types &&\n> +\tprintf \"blob\\ntree\\n\" > expected &&\n> +\ttest_cmp filtered_types expected\n> +'\n> +\n> +\n> # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n> # This models previously omitted objects that we did not receive.\n> \n> \n"},{"id":"355555","messageId":"20180814160303.693-1-jonathantanmy@google.com","threadId":"49086","inReplyTo":"CAMfpvh+awO8kpVtxKxMDuiXx2XM0o3xGycXdSF+ZJ0NXyzv5Hg@mail.gmail.com","subject":"Re: [PATCH v2 3/5] rev-list: handle missing tree objects properly","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-08-14T16:03:03Z","receivedAt":"2018-08-14T16:03:09Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> I don't understand about the \">= 0\". What should I replace it with?\n> Maybe you mean the return is never positive so I can change:\n> \n> parse_tree_gently(tree, 1) >= 0\n> \n> to:\n> !parse_tree_gently(tree, 1)\n> \n> ?\n\nSorry for the lack of clarity - that is what I meant.\n\n> > The missing mechanism (for error, allow-any, print) should work without\n> > needing to consult whether an object is a promisor object or not - it\n> > should just print whatever is missing, so the \"if\n> > (!is_promisor_object...\" line looks out of place.\n> Done. I considered that a missing object which is not a promisor is a\n> serious error, so I had it die here.\n\nIt is a serious error, but as far as I can tell, that is what the\n--missing flags are supposed to help diagnose (so we can't die since we\nneed the diagnoses to be printed). See, for example, 'rev-list W/\n--missing=print' in t6112 - the \"r1\" repository does not have partial\nclone enabled (which I verified by inserting a test_pause then cat-ting\nr1/.git/config), but nothing dies.\n\n> But now that I've added the\n> do_not_die_on_missing_tree flag, it's more natural to keep the\n> previous promisor check as-is.\n\nOK, I'll take a look once you send out v4.\n\n> Also, is_promisor_object is an\n> expensive check, and it would be better to skip it during the common\n> execution path (which should be when exclude_promisor_objects, an\n> internal-use-only flag, is *not* set, which means we never call\n> is_promisor_object.\n\nThat's true.\n\n> > In my original review [1], I suggested that we always show a tree if we\n> > have its hash - if we don't have the object, we just recurse into it.\n> > This would be the same as your patch, except that the 'die(\"bad tree\n> > object...' is totally removed instead of merely moved. I still think\n> > this solution has some merit - all the tests still pass (except that we\n> > need to check for \"unable to read\" instead of \"bad tree object\" in error\n> > messages), but I just realized that it might still be backwards\n> > incompatible in that a basic \"rev-list --objects\" would now succeed\n> > instead of fail if a tree was missing (I haven't tested this though).\n> The presence of the die if !is_promisor_object is what justified the\n> changing of the parse_tree_gently to always be gently, since it is\n> what showed the OID. Can we really remove both? Maybe in a different\n> patch set, since I'm no longer touching that line?\n\nThat's true - the idea of removing both needs more thought, and if we\nwere to do so, we definitely can do it in a different patch set.\n\n> > We might need a flag called \"do_not_die_on_missing_tree\" (much like your\n> > original idea of \"show_missing_trees\") so that callers that are prepared\n> > to deal with missing trees can set this. Sorry for the churn. You can\n> > document it as such:\n> Added it, but not with a command-line flag, only in rev-info.h. We can\n> always add a flag later if people have been relying on the existing\n> behavior of git rev-list to balk at missing trees. (That seems\n> unlikely though, considering there is no filter to enable that before\n> this patchset).\n\nBy flag, I indeed meant in rev-info.h - sorry for the confusion. That\nsounds good.\n"},{"id":"355563","messageId":"c54a2818125739b7dde40aeee2cf1d72e057e963.1534267611.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534267611.git.matvore@google.com","subject":"[PATCH v4 1/6] list-objects: store common func args in struct","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-14T17:28:08Z","receivedAt":"2018-08-14T17:29:12Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will make utility functions easier to create, as done by the next\npatch.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 158 +++++++++++++++++++++++--------------------------\n 1 file changed, 74 insertions(+), 84 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex c99c47ac1..584518a3f 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -12,20 +12,25 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n \n-static void process_blob(struct rev_info *revs,\n+struct traversal_context {\n+\tstruct rev_info *revs;\n+\tshow_object_fn show_object;\n+\tshow_commit_fn show_commit;\n+\tvoid *show_data;\n+\tfilter_object_fn filter_fn;\n+\tvoid *filter_data;\n+};\n+\n+static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \n-\tif (!revs->blob_objects)\n+\tif (!ctx->revs->blob_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad blob object\");\n@@ -41,21 +46,21 @@ static void process_blob(struct rev_info *revs,\n \t * may cause the actual filter to report an incomplete list\n \t * of missing objects.\n \t */\n-\tif (revs->exclude_promisor_objects &&\n+\tif (ctx->revs->exclude_promisor_objects &&\n \t !has_object_file(&obj->oid) &&\n \t is_promisor_object(&obj->oid))\n \t\treturn;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BLOB, obj,\n-\t\t\t path->buf, &path->buf[pathlen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n+\t\t\t\t path->buf, &path->buf[pathlen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, path->buf, cb_data);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -81,26 +86,21 @@ static void process_blob(struct rev_info *revs,\n * the link, and how to do it. Whether it necessarily makes\n * any sense what-so-ever to ever do that is another issue.\n */\n-static void process_gitlink(struct rev_info *revs,\n+static void process_gitlink(struct traversal_context *ctx,\n \t\t\t const unsigned char *sha1,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data)\n+\t\t\t const char *name)\n {\n \t/* Nothing to do */\n }\n \n-static void process_tree(struct rev_info *revs,\n+static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &tree->object;\n+\tstruct rev_info *revs = ctx->revs;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n@@ -133,14 +133,14 @@ static void process_tree(struct rev_info *revs,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BEGIN_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, base->buf, cb_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -157,29 +157,25 @@ static void process_tree(struct rev_info *revs,\n \t\t}\n \n \t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(revs,\n+\t\t\tprocess_tree(ctx,\n \t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(revs, entry.oid->hash,\n-\t\t\t\t\tshow, base, entry.path,\n-\t\t\t\t\tcb_data);\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n \t\telse\n-\t\t\tprocess_blob(revs,\n+\t\t\tprocess_blob(ctx,\n \t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t}\n \n-\tif (!(obj->flags & USER_GIVEN) && filter_fn) {\n-\t\tr = filter_fn(LOFS_END_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \t\tif (r & LOFR_MARK_SEEN)\n \t\t\tobj->flags |= SEEN;\n \t\tif (r & LOFR_DO_SHOW)\n-\t\t\tshow(obj, base->buf, cb_data);\n+\t\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \t}\n \n \tstrbuf_setlen(base, baselen);\n@@ -242,19 +238,15 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n-static void traverse_trees_and_blobs(struct rev_info *revs,\n-\t\t\t\t struct strbuf *base,\n-\t\t\t\t show_object_fn show_object,\n-\t\t\t\t void *show_data,\n-\t\t\t\t filter_object_fn filter_fn,\n-\t\t\t\t void *filter_data)\n+static void traverse_trees_and_blobs(struct traversal_context *ctx,\n+\t\t\t\t struct strbuf *base)\n {\n \tint i;\n \n \tassert(base->len == 0);\n \n-\tfor (i = 0; i < revs->pending.nr; i++) {\n-\t\tstruct object_array_entry *pending = revs->pending.objects + i;\n+\tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n+\t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n@@ -262,62 +254,49 @@ static void traverse_trees_and_blobs(struct rev_info *revs,\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tshow_object(obj, name, show_data);\n+\t\t\tctx->show_object(obj, name, ctx->show_data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(revs, (struct tree *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(revs, (struct blob *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n \t\t oid_to_hex(&obj->oid), name);\n \t}\n-\tobject_array_clear(&revs->pending);\n+\tobject_array_clear(&ctx->revs->pending);\n }\n \n-static void do_traverse(struct rev_info *revs,\n-\t\t\tshow_commit_fn show_commit,\n-\t\t\tshow_object_fn show_object,\n-\t\t\tvoid *show_data,\n-\t\t\tfilter_object_fn filter_fn,\n-\t\t\tvoid *filter_data)\n+static void do_traverse(struct traversal_context *ctx)\n {\n \tstruct commit *commit;\n \tstruct strbuf csp; /* callee's scratch pad */\n \tstrbuf_init(&csp, PATH_MAX);\n \n-\twhile ((commit = get_revision(revs)) != NULL) {\n+\twhile ((commit = get_revision(ctx->revs)) != NULL) {\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n \t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(revs, get_commit_tree(commit));\n-\t\tshow_commit(commit, show_data);\n+\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tctx->show_commit(commit, ctx->show_data);\n \n-\t\tif (revs->tree_blobs_in_commit_order)\n+\t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n \t\t\t * NEEDSWORK: Adding the tree and then flushing it here\n \t\t\t * needs a reallocation for each commit. Can we pass the\n \t\t\t * tree directory without allocation churn?\n \t\t\t */\n-\t\t\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t\t\t show_object, show_data,\n-\t\t\t\t\t\t filter_fn, filter_data);\n+\t\t\ttraverse_trees_and_blobs(ctx, &csp);\n \t}\n-\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t show_object, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\ttraverse_trees_and_blobs(ctx, &csp);\n \tstrbuf_release(&csp);\n }\n \n@@ -326,7 +305,14 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t\t show_object_fn show_object,\n \t\t\t void *show_data)\n {\n-\tdo_traverse(revs, show_commit, show_object, show_data, NULL, NULL);\n+\tstruct traversal_context ctx;\n+\tctx.revs = revs;\n+\tctx.show_commit = show_commit;\n+\tctx.show_object = show_object;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\tctx.filter_data = NULL;\n+\tdo_traverse(&ctx);\n }\n \n void traverse_commit_list_filtered(\n@@ -337,14 +323,18 @@ void traverse_commit_list_filtered(\n \tvoid *show_data,\n \tstruct oidset *omitted)\n {\n-\tfilter_object_fn filter_fn = NULL;\n+\tstruct traversal_context ctx;\n \tfilter_free_fn filter_free_fn = NULL;\n-\tvoid *filter_data = NULL;\n-\n-\tfilter_data = list_objects_filter__init(omitted, filter_options,\n-\t\t\t\t\t\t&filter_fn, &filter_free_fn);\n-\tdo_traverse(revs, show_commit, show_object, show_data,\n-\t\t filter_fn, filter_data);\n-\tif (filter_data && filter_free_fn)\n-\t\tfilter_free_fn(filter_data);\n+\n+\tctx.revs = revs;\n+\tctx.show_object = show_object;\n+\tctx.show_commit = show_commit;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\n+\tctx.filter_data = list_objects_filter__init(omitted, filter_options,\n+\t\t\t\t\t\t &ctx.filter_fn, &filter_free_fn);\n+\tdo_traverse(&ctx);\n+\tif (ctx.filter_data && filter_free_fn)\n+\t\tfilter_free_fn(ctx.filter_data);\n }\n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355564","messageId":"feb7ea58228df4bb777ee96651389000e804f9fa.1534267611.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534267611.git.matvore@google.com","subject":"[PATCH v4 2/6] list-objects: refactor to process_tree_contents","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-14T17:28:09Z","receivedAt":"2018-08-14T17:29:16Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will be used in a follow-up patch to reduce indentation needed when\ninvoking the logic conditionally. i.e. rather than:\n\nif (foo) {\n\twhile (...) {\n\t\t/* this is very indented */\n\t}\n}\n\nwe will have:\n\nif (foo)\n\tprocess_tree_contents(...);\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 68 ++++++++++++++++++++++++++++++--------------------\n 1 file changed, 41 insertions(+), 27 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 584518a3f..ccc529e5e 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -94,6 +94,46 @@ static void process_gitlink(struct traversal_context *ctx,\n \t/* Nothing to do */\n }\n \n+static void process_tree(struct traversal_context *ctx,\n+\t\t\t struct tree *tree,\n+\t\t\t struct strbuf *base,\n+\t\t\t const char *name);\n+\n+static void process_tree_contents(struct traversal_context *ctx,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *base)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tenum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ?\n+\t\tall_entries_interesting : entry_not_interesting;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tif (match != all_entries_interesting) {\n+\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n+\t\t\t\t\t\t &ctx->revs->diffopt.pathspec);\n+\t\t\tif (match == all_entries_not_interesting)\n+\t\t\t\tbreak;\n+\t\t\tif (match == entry_not_interesting)\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (S_ISDIR(entry.mode))\n+\t\t\tprocess_tree(ctx,\n+\t\t\t\t lookup_tree(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t\telse if (S_ISGITLINK(entry.mode))\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash,\n+\t\t\t\t\tbase, entry.path);\n+\t\telse\n+\t\t\tprocess_blob(ctx,\n+\t\t\t\t lookup_blob(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t}\n+}\n+\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n@@ -101,10 +141,6 @@ static void process_tree(struct traversal_context *ctx,\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n-\tstruct tree_desc desc;\n-\tstruct name_entry entry;\n-\tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n-\t\tall_entries_interesting: entry_not_interesting;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \tint gently = revs->ignore_missing_links ||\n@@ -144,29 +180,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tinit_tree_desc(&desc, tree->buffer, tree->size);\n-\n-\twhile (tree_entry(&desc, &entry)) {\n-\t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n-\t\t\t\t\t\t &revs->diffopt.pathspec);\n-\t\t\tif (match == all_entries_not_interesting)\n-\t\t\t\tbreak;\n-\t\t\tif (match == entry_not_interesting)\n-\t\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t}\n+\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355565","messageId":"e01e202cb0445b72dc52a1c8763dbd9c737fb86e.1534267611.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534267611.git.matvore@google.com","subject":"[PATCH v4 4/6] rev-list: handle missing tree objects properly","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-14T17:28:11Z","receivedAt":"2018-08-14T17:29:20Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Previously, we assumed only blob objects could be missing. This patch\nmakes rev-list handle missing trees like missing blobs. A missing tree\nwill cause an error if --missing indicates an error should be caused,\nand the hash is printed even if the tree is missing.\n\nIn list-objects.c we no longer print a message to stderr if a tree\nobject is missing (quiet_on_missing is always true). I couldn't find\nany place where this would matter, or where the caller of\ntraverse_commit_list would need to be fixed to show the error. However,\nin the future it would be trivial to make the caller show the message if\nwe needed to.\n\nThis is not tested very thoroughly, since we cannot create promisor\nobjects in tests without using an actual partial clone. t0410 has a\npromise_and_delete utility function, but the is_promisor_object function\ndoes not return 1 for objects deleted in this way. More tests will will\ncome in a patch that implements a filter that can be used with git\nclone.\n\nSigned-off-by: Matthew DeVore \n---\n builtin/rev-list.c | 11 +++--\n list-objects.c | 17 +++++--\n revision.h | 1 +\n t/t0410-partial-clone.sh | 66 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 13 +++++\n 5 files changed, 101 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 5b07f3f4a..49d6deed7 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -6,6 +6,7 @@\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"object.h\"\n #include \"object-store.h\"\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n@@ -209,7 +210,8 @@ static inline void finish_object__ma(struct object *obj)\n \t */\n \tswitch (arg_missing_action) {\n \tcase MA_ERROR:\n-\t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n+\t\tdie(\"missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tcase MA_ALLOW_ANY:\n@@ -222,8 +224,8 @@ static inline void finish_object__ma(struct object *obj)\n \tcase MA_ALLOW_PROMISOR:\n \t\tif (is_promisor_object(&obj->oid))\n \t\t\treturn;\n-\t\tdie(\"unexpected missing blob object '%s'\",\n-\t\t oid_to_hex(&obj->oid));\n+\t\tdie(\"unexpected missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tdefault:\n@@ -235,7 +237,7 @@ static inline void finish_object__ma(struct object *obj)\n static int finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\tif (!has_object_file(&obj->oid)) {\n \t\tfinish_object__ma(obj);\n \t\treturn 1;\n \t}\n@@ -373,6 +375,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tinit_revisions(&revs, prefix);\n \trevs.abbrev = DEFAULT_ABBREV;\n \trevs.commit_format = CMIT_FMT_UNSPECIFIED;\n+\trevs.do_not_die_on_missing_tree = 1;\n \n \t/*\n \t * Scan the argument list before invoking setup_revisions(), so that we\ndiff --git a/list-objects.c b/list-objects.c\nindex f9b51db7a..e88474a2d 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,6 +143,7 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tint parsed;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -150,7 +151,9 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, 1) < 0) {\n+\n+\tparsed = parse_tree_gently(tree, 1) >= 0;\n+\tif (!parsed) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n@@ -163,7 +166,8 @@ static void process_tree(struct traversal_context *ctx,\n \t\t is_promisor_object(&obj->oid))\n \t\t\treturn;\n \n-\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n+\t\tif (!revs->do_not_die_on_missing_tree)\n+\t\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \n \tstrbuf_addstr(base, name);\n@@ -178,7 +182,14 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tprocess_tree_contents(ctx, tree, base);\n+\t/*\n+\t * NEEDSWORK: we should not have to process this tree's contents if the\n+\t * filter wants to exclude all its contents AND the filter doesn't need\n+\t * to collect the omitted OIDs. We should add a LOFR_SKIP_TREE bit which\n+\t * allows skipping all children.\n+\t */\n+\tif (parsed)\n+\t\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\ndiff --git a/revision.h b/revision.h\nindex c599c34da..c94243543 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -124,6 +124,7 @@ struct rev_info {\n \t\t\tfirst_parent_only:1,\n \t\t\tline_level_traverse:1,\n \t\t\ttree_blobs_in_commit_order:1,\n+\t\t\tdo_not_die_on_missing_tree:1,\n \n \t\t\t/* for internal use only */\n \t\t\texclude_promisor_objects:1;\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 4984ca583..74e3c5767 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -186,6 +186,72 @@ test_expect_success 'rev-list stops traversal at missing and promised commit' '\n \t! grep $FOO out\n '\n \n+test_expect_success 'show missing tree objects with --missing=print' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo foo &&\n+\ttest_commit -C repo bar &&\n+\ttest_commit -C repo baz &&\n+\n+\tTREE=$(git -C repo rev-parse bar^{tree}) &&\n+\n+\tpromise_and_delete $TREE &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo rev-list --quiet --missing=print --objects HEAD >missing_objs 2>rev_list_err &&\n+\techo \"?$TREE\" >expected &&\n+\ttest_cmp expected missing_objs &&\n+\n+\t# do not complain when a missing tree cannot be parsed\n+\t! grep -q \"Could not read \" rev_list_err\n+'\n+\n+test_expect_success 'missing tree objects with --missing=allow-any and --exclude-promisor-objects' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo foo &&\n+\ttest_commit -C repo bar &&\n+\ttest_commit -C repo baz &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse bar^{tree}) &&\n+\tpromise_and_delete $(git -C repo rev-parse foo^{tree}) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n+\ttest_line_count = 0 rev_list_err &&\n+\t# 3 commits, 3 blobs, and 1 tree\n+\ttest_line_count = 7 objs &&\n+\n+\t# Do the same for --exclude-promisor-objects, but with all trees gone.\n+\tpromise_and_delete $(git -C repo rev-parse baz^{tree}) &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >objs 2>rev_list_err &&\n+\ttest_line_count = 0 rev_list_err &&\n+\t# 3 commits, no blobs or trees\n+\ttest_line_count = 3 objs\n+'\n+\n+test_expect_success 'missing non-root tree object and rev-list' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\tmkdir repo/dir &&\n+\techo foo > repo/dir/foo &&\n+\tgit -C repo add dir/foo &&\n+\tgit -C repo commit -m \"commit dir/foo\" &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse HEAD:dir) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n+\ttest_line_count = 0 rev_list_err &&\n+\t# 1 commit and 1 tree\n+\ttest_line_count = 2 objs\n+'\n+\n test_expect_success 'rev-list stops traversal at missing and promised tree' '\n \trm -rf repo &&\n \ttest_create_repo repo &&\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 6710c8bc8..5e35f33bf 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -59,6 +59,19 @@ test_expect_success 'verify normal and blob:none packfiles have same commits/tre\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'get an error for missing tree object' '\n+\tgit init r5 &&\n+\techo foo > r5/foo &&\n+\tgit -C r5 add foo &&\n+\tgit -C r5 commit -m \"foo\" &&\n+\tdel=$(git -C r5 rev-parse HEAD^{tree} | sed \"s|..|&/|\") &&\n+\trm r5/.git/objects/$del &&\n+\ttest_must_fail git -C r5 pack-objects --rev --stdout 2>bad_tree <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgrep -q \"bad tree object\" bad_tree\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355566","messageId":"7b9ffe47c4e91d442078e1ed99b53653f5fed6a3.1534267611.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534267611.git.matvore@google.com","subject":"[PATCH v4 3/6] list-objects: always parse trees gently","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-14T17:28:10Z","receivedAt":"2018-08-14T17:29:22Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"If parsing fails when revs->ignore_missing_links and\nrevs->exclude_promisor_objects are both false, we print the OID anyway\nin the die(\"bad tree object...\") call, so any message printed by\nparse_tree_gently() is superfluous.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex ccc529e5e..f9b51db7a 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,8 +143,6 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n-\tint gently = revs->ignore_missing_links ||\n-\t\t revs->exclude_promisor_objects;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -152,7 +150,7 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, gently) < 0) {\n+\tif (parse_tree_gently(tree, 1) < 0) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355567","messageId":"a3284bac2d6c317347ed74d43f474b3998050fd1.1534267611.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534267611.git.matvore@google.com","subject":"[PATCH v4 5/6] revision: mark non-user-given objects instead","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-14T17:28:12Z","receivedAt":"2018-08-14T17:29:22Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Currently, list-objects.c incorrectly treats all root trees of commits\nas USER_GIVEN. Also, it would be easier to mark objects that are\nnon-user-given instead of user-given, since the places in the code\nwhere we access an object through a reference are more obvious than\nthe places where we access an object that was given by the user.\n\nResolve these two problems by introducing a flag NOT_USER_GIVEN that\nmarks blobs and trees that are non-user-given, replacing USER_GIVEN.\n(Only blobs and trees are marked because this mark is only used when\nfiltering objects, and filtering of other types of objects is not\nsupported yet.)\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 31 ++++++++++++++++++-------------\n revision.c | 1 -\n revision.h | 10 +++++++---\n 3 files changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex e88474a2d..3aee1b8a4 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -53,7 +53,7 @@ static void process_blob(struct traversal_context *ctx,\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n \t\t\t\t path->buf, &path->buf[pathlen],\n \t\t\t\t ctx->filter_data);\n@@ -120,17 +120,19 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\tcontinue;\n \t\t}\n \n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\tstruct tree *t = lookup_tree(the_repository, entry.oid);\n+\t\t\tt->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_tree(ctx, t, base, entry.path);\n+\t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid->hash,\n \t\t\t\t\tbase, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\telse {\n+\t\t\tstruct blob *b = lookup_blob(the_repository, entry.oid);\n+\t\t\tb->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_blob(ctx, b, base, entry.path);\n+\t\t}\n \t}\n }\n \n@@ -171,7 +173,7 @@ static void process_tree(struct traversal_context *ctx,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -191,7 +193,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (parsed)\n \t\tprocess_tree_contents(ctx, tree, base);\n \n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -307,8 +309,11 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n-\t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tif (get_commit_tree(commit)) {\n+\t\t\tstruct tree *tree = get_commit_tree(commit);\n+\t\t\ttree->object.flags |= NOT_USER_GIVEN;\n+\t\t\tadd_pending_tree(ctx->revs, tree);\n+\t\t}\n \t\tctx->show_commit(commit, ctx->show_data);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\ndiff --git a/revision.c b/revision.c\nindex 062749437..6d355b43c 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -175,7 +175,6 @@ static void add_pending_object_with_path(struct rev_info *revs,\n \t\tstrbuf_release(&buf);\n \t\treturn; /* do not add the commit itself */\n \t}\n-\tobj->flags |= USER_GIVEN;\n \tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n }\n \ndiff --git a/revision.h b/revision.h\nindex c94243543..55d47004d 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -8,7 +8,11 @@\n #include \"diff.h\"\n #include \"commit-slab-decl.h\"\n \n-/* Remember to update object flag allocation in object.h */\n+/* Remember to update object flag allocation in object.h\n+ * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n+ * filtering trees and blobs, but it may be useful to support filtering commits\n+ * in the future.\n+ */\n #define SEEN\t\t(1u<<0)\n #define UNINTERESTING (1u<<1)\n #define TREESAME\t(1u<<2)\n@@ -20,9 +24,9 @@\n #define SYMMETRIC_LEFT\t(1u<<8)\n #define PATCHSAME\t(1u<<9)\n #define BOTTOM\t\t(1u<<10)\n-#define USER_GIVEN\t(1u<<25) /* given directly by the user */\n+#define NOT_USER_GIVEN\t(1u<<25) /* tree or blob not given directly by user */\n #define TRACK_LINEAR\t(1u<<26)\n-#define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n+#define ALL_REV_FLAGS\t(((1u<<11)-1) | NOT_USER_GIVEN | TRACK_LINEAR)\n \n #define DECORATE_SHORT_REFS\t1\n #define DECORATE_FULL_REFS\t2\n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355571","messageId":"20180814180626.31788-1-jonathantanmy@google.com","threadId":"49086","inReplyTo":"e01e202cb0445b72dc52a1c8763dbd9c737fb86e.1534267611.git.matvore@google.com","subject":"Re: [PATCH v4 4/6] rev-list: handle missing tree objects properly","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-08-14T18:06:26Z","receivedAt":"2018-08-14T18:06:33Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Previously, we assumed only blob objects could be missing. This patch\n> makes rev-list handle missing trees like missing blobs. A missing tree\n> will cause an error if --missing indicates an error should be caused,\n> and the hash is printed even if the tree is missing.\n\nThe last sentence is difficult to understand - probably better to say\nthat all --missing= arguments and --exclude-promisor-objects work for\nmissing trees like they currently do for blobs (and do not fixate on\njust --missing=error). And also demonstrate this in tests, like in\nt6612.\n\n> In list-objects.c we no longer print a message to stderr if a tree\n> object is missing (quiet_on_missing is always true). I couldn't find\n> any place where this would matter, or where the caller of\n> traverse_commit_list would need to be fixed to show the error. However,\n> in the future it would be trivial to make the caller show the message if\n> we needed to.\n> \n> This is not tested very thoroughly, since we cannot create promisor\n> objects in tests without using an actual partial clone. t0410 has a\n> promise_and_delete utility function, but the is_promisor_object function\n> does not return 1 for objects deleted in this way. More tests will will\n> come in a patch that implements a filter that can be used with git\n> clone.\n\nThese two paragraphs are no longer applicable, I think.\n\n> @@ -373,6 +375,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n> \tinit_revisions(&revs, prefix);\n> \trevs.abbrev = DEFAULT_ABBREV;\n> \trevs.commit_format = CMIT_FMT_UNSPECIFIED;\n> +\trevs.do_not_die_on_missing_tree = 1;\n\nIs this correct? I would have expected this to be set only if --missing\nwas set.\n\n> -\tprocess_tree_contents(ctx, tree, base);\n> +\t/*\n> +\t * NEEDSWORK: we should not have to process this tree's contents if the\n> +\t * filter wants to exclude all its contents AND the filter doesn't need\n> +\t * to collect the omitted OIDs. We should add a LOFR_SKIP_TREE bit which\n> +\t * allows skipping all children.\n> +\t */\n> +\tif (parsed)\n> +\t\tprocess_tree_contents(ctx, tree, base);\n\nI agree with Jeff Hostetler in [1] that a LOFR_SKIP_TREE bit is\ndesirable, but I don't think that this patch is the right place to\nintroduce this NEEDSWORK. For me, this patch is about skipping iterating\nover the contents of a tree because the tree does not exist; this\nNEEDSWORK is about skipping iterating over the contents of a tree\nbecause we don't want its contents, and it is quite confusing to\nconflate the two.\n\n[1] https://public-inbox.org/git/d751d56b-84bb-a03d-5f2a-7dbaf8d947cc@jeffhostetler.com/\n\n> @@ -124,6 +124,7 @@ struct rev_info {\n> \t\t\tfirst_parent_only:1,\n> \t\t\tline_level_traverse:1,\n> \t\t\ttree_blobs_in_commit_order:1,\n> +\t\t\tdo_not_die_on_missing_tree:1,\n\nI know that the other flags don't have documentation, but I think it's\nworth documenting this one because it is rather complicated. I have\nprovided a sample one in my earlier review - feel free to use that or\ncome up with your own.\n\n> diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\n> index 4984ca583..74e3c5767 100755\n> --- a/t/t0410-partial-clone.sh\n> +++ b/t/t0410-partial-clone.sh\n> @@ -186,6 +186,72 @@ test_expect_success 'rev-list stops traversal at missing and promised commit' '\n> \t! grep $FOO out\n> '\n> \n> +test_expect_success 'show missing tree objects with --missing=print' '\n> +\trm -rf repo &&\n> +\ttest_create_repo repo &&\n> +\ttest_commit -C repo foo &&\n> +\ttest_commit -C repo bar &&\n> +\ttest_commit -C repo baz &&\n> +\n> +\tTREE=$(git -C repo rev-parse bar^{tree}) &&\n> +\n> +\tpromise_and_delete $TREE &&\n> +\n> +\tgit -C repo config core.repositoryformatversion 1 &&\n> +\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n> +\tgit -C repo rev-list --quiet --missing=print --objects HEAD >missing_objs 2>rev_list_err &&\n> +\techo \"?$TREE\" >expected &&\n> +\ttest_cmp expected missing_objs &&\n> +\n> +\t# do not complain when a missing tree cannot be parsed\n> +\t! grep -q \"Could not read \" rev_list_err\n> +'\n\nI think that the --exclude-promisor-tests can go in t0410 as you have\ndone, but the --missing tests (except for --missing=allow-promisor)\nshould go in t6112. (And like the existing --missing tests, they should\nbe done without setting extensions.partialclone.)\n\nAs for --missing=allow-promisor, I don't see them being tested anywhere\n:-( so feel free to make a suggestion. I would put them in t6112 for\neasy comparison with the other --missing tests.\n"},{"id":"355572","messageId":"05e69ef260ca2b28b2444b94850bddf44ca0388a.1534267611.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534267611.git.matvore@google.com","subject":"[PATCH v4 6/6] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-14T17:28:13Z","receivedAt":"2018-08-14T18:09:06Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Teach list-objects the \"tree:0\" filter which allows for filtering\nout all tree and blob objects (unless other objects are explicitly\nspecified by the user). The purpose of this patch is to allow smaller\npartial clones.\n\nThe name of this filter - tree:0 - does not explicitly specify that\nit also filters out all blobs, but this should not cause much confusion\nbecause blobs are not at all useful without the trees that refer to\nthem.\n\nI also consider only:commits as a name, but this is inaccurate because\nit suggests that annotated tags are omitted, but actually they are\nincluded.\n\nThe name \"tree:0\" allows later filtering based on depth, i.e. \"tree:1\"\nwould filter out all but the root tree and blobs. In order to avoid\nconfusion between 0 and capital O, the documentation was worded in a\nsomewhat round-about way that also hints at this future improvement to\nthe feature.\n\nSigned-off-by: Matthew DeVore \n---\n Documentation/rev-list-options.txt | 3 ++\n list-objects-filter-options.c | 4 +++\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 50 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 27 ++++++++++++++\n t/t5616-partial-clone.sh | 38 ++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 13 +++++++\n 7 files changed, 136 insertions(+)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 7b273635d..9e351ec2a 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -743,6 +743,9 @@ specification contained in .\n \tA debug option to help with future \"partial clone\" development.\n \tThis option specifies how missing objects are handled.\n +\n+The form '--filter=tree:' omits all blobs and trees deeper than\n+ from the root tree. Currently, only =0 is supported.\n++\n The form '--missing=error' requests that rev-list stop with an error if\n a missing object is encountered. This is the default action.\n +\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex c0e2bd6a0..a28382940 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -50,6 +50,10 @@ static int gently_parse_list_objects_filter(\n \t\t\treturn 0;\n \t\t}\n \n+\t} else if (!strcmp(arg, \"tree:0\")) {\n+\t\tfilter_options->choice = LOFC_TREE_NONE;\n+\t\treturn 0;\n+\n \t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n \t\tstruct object_context oc;\n \t\tstruct object_id sparse_oid;\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 0000a61f8..af64e5c66 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -10,6 +10,7 @@ enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n+\tLOFC_TREE_NONE,\n \tLOFC_SPARSE_OID,\n \tLOFC_SPARSE_PATH,\n \tLOFC__COUNT /* must be last */\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex a0ba78b20..8e3caf5bf 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -80,6 +80,55 @@ static void *filter_blobs_none__init(\n \treturn d;\n }\n \n+/*\n+ * A filter for list-objects to omit ALL trees and blobs from the traversal.\n+ * Can OPTIONALLY collect a list of the omitted OIDs.\n+ */\n+struct filter_trees_none_data {\n+\tstruct oidset *omits;\n+};\n+\n+static enum list_objects_filter_result filter_trees_none(\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_trees_none_data *filter_data = filter_data_;\n+\n+\tswitch (filter_situation) {\n+\tdefault:\n+\t\tdie(\"unknown filter_situation\");\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFS_BEGIN_TREE:\n+\tcase LOFS_BLOB:\n+\t\tif (filter_data->omits)\n+\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n+\n+\tcase LOFS_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\treturn LOFR_ZERO;\n+\n+\t}\n+}\n+\n+static void* filter_trees_none__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn)\n+{\n+\tstruct filter_trees_none_data *d = xcalloc(1, sizeof(*d));\n+\td->omits = omitted;\n+\n+\t*filter_fn = filter_trees_none;\n+\t*filter_free_fn = free;\n+\treturn d;\n+}\n+\n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n@@ -374,6 +423,7 @@ static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n+\tfilter_trees_none__init,\n \tfilter_sparse_oid__init,\n \tfilter_sparse_path__init,\n };\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 5e35f33bf..65f2cf446 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -72,6 +72,33 @@ test_expect_success 'get an error for missing tree object' '\n \tgrep -q \"bad tree object\" bad_tree\n '\n \n+test_expect_success 'setup for tests of tree:0' '\n+\tmkdir r1/subtree &&\n+\techo \"This is a file in a subtree\" > r1/subtree/file &&\n+\tgit -C r1 add subtree/file &&\n+\tgit -C r1 commit -m subtree\n+'\n+\n+test_expect_success 'verify tree:0 packfile has no blobs or trees' '\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\t! grep -E \"tree|blob\" objs\n+'\n+\n+test_expect_success 'grab tree directly when using tree:0' '\n+\t# We should get the tree specified directly but not its blobs or subtrees.\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD:\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\tgrep -E \"tree|blob\" objs >trees_and_blobs &&\n+\ttest_line_count = 1 trees_and_blobs\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex bbbe7537d..d2859aba1 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -154,6 +154,22 @@ test_expect_success 'partial clone with transfer.fsckobjects=1 uses index-pack -\n \tgrep \"git index-pack.*--fsck-objects\" trace\n '\n \n+test_expect_success 'use fsck before and after manually fetching a missing subtree' '\n+\t# push new commit so server has a subtree\n+\tmkdir src/dir &&\n+\techo \"in dir\" > src/dir/file.txt &&\n+\tgit -C src add dir/file.txt &&\n+\tgit -C src commit -m \"file in dir\" &&\n+\tgit -C src push -u srv master &&\n+\tSUBTREE=$(git -C src rev-parse HEAD:dir) &&\n+\n+\trm -rf dst &&\n+\tgit clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n+\tgit -C dst fsck &&\n+\tgit -C dst cat-file -p $SUBTREE >tree_contents 2>err &&\n+\tgit -C dst fsck\n+'\n+\n test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' '\n \trm -rf src dst &&\n \tgit init src &&\n@@ -170,6 +186,28 @@ test_expect_success 'partial clone fetches blobs pointed to by refs even if norm\n \tgit -C dst fsck\n '\n \n+test_expect_success 'can use tree:0 to filter partial clone' '\n+\trm -rf dst &&\n+\tgit clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n+\tgit -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n+\tcat fetched_objects \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n+\tsort fetched_types -u >unique_types.observed &&\n+\techo commit > unique_types.expected &&\n+\ttest_cmp unique_types.observed unique_types.expected\n+'\n+\n+test_expect_success 'auto-fetching of trees with --missing=error' '\n+\tgit -C dst rev-list master --missing=error --objects >fetched_objects &&\n+\tcat fetched_objects \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n+\tsort fetched_types -u >unique_types.observed &&\n+\tprintf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n+\ttest_cmp unique_types.observed unique_types.expected\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 0a37dd5f9..6ccffddbc 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -196,6 +196,19 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n \ttest_cmp observed expected\n '\n \n+# Test tree:0 filter.\n+\n+test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n+\tgit -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=tree:0 \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed s/~// \\\n+\t\t| xargs -n1 git -C r3 cat-file -t \\\n+\t\t| sort -u >filtered_types &&\n+\tprintf \"blob\\ntree\\n\" > expected &&\n+\ttest_cmp filtered_types expected\n+'\n+\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355573","messageId":"cover.1534267611.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"[PATCH v4 0/6] filter: support for excluding all trees and blobs","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-14T17:28:07Z","receivedAt":"2018-08-14T18:09:27Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"I've applied or responded to all changes suggested by Jonathan and Jeff.\n\nMatthew DeVore (6):\n list-objects: store common func args in struct\n list-objects: refactor to process_tree_contents\n list-objects: always parse trees gently\n rev-list: handle missing tree objects properly\n revision: mark non-user-given objects instead\n list-objects-filter: implement filter tree:0\n\n Documentation/rev-list-options.txt | 3 +\n builtin/rev-list.c | 11 +-\n list-objects-filter-options.c | 4 +\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 50 ++++++\n list-objects.c | 238 +++++++++++++------------\n revision.c | 1 -\n revision.h | 11 +-\n t/t0410-partial-clone.sh | 66 +++++++\n t/t5317-pack-objects-filter-objects.sh | 40 +++++\n t/t5616-partial-clone.sh | 38 ++++\n t/t6112-rev-list-filters-objects.sh | 13 ++\n 12 files changed, 358 insertions(+), 118 deletions(-)\n\n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355574","messageId":"CAMfpvhLevPWundgHHuMUE+x019iVjpcfR-GgtTPvkHwpuXpH3w@mail.gmail.com","threadId":"49086","inReplyTo":"d751d56b-84bb-a03d-5f2a-7dbaf8d947cc@jeffhostetler.com","subject":"Re: [PATCH v3 5/5] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-14T17:25:33Z","receivedAt":"2018-08-14T18:12:42Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Tue, Aug 14, 2018 at 8:13 AM Jeff Hostetler wrote:\n>\n>\n>\n> On 8/13/2018 2:14 PM, Matthew DeVore wrote:\n> > Teach list-objects the \"tree:0\" filter which allows for filtering\n> > out all tree and blob objects (unless other objects are explicitly\n> > specified by the user). The purpose of this patch is to allow smaller\n> > partial clones.\n> >\n> > The name of this filter - tree:0 - does not explicitly specify that\n> > it also filters out all blobs, but this should not cause much confusion\n> > because blobs are not at all useful without the trees that refer to\n> > them.\n> >\n> > I also consider only:commits as a name, but this is inaccurate because\n> > it suggests that annotated tags are omitted, but actually they are\n> > included.\n> >\n> > The name \"tree:0\" allows later filtering based on depth, i.e. \"tree:1\"\n> > would filter out all but the root tree and blobs. In order to avoid\n> > confusion between 0 and capital O, the documentation was worded in a\n> > somewhat round-about way that also hints at this future improvement to\n> > the feature.\n> >\n> > Signed-off-by: Matthew DeVore \n> > ---\n> > Documentation/rev-list-options.txt | 3 ++\n> > list-objects-filter-options.c | 4 +++\n> > list-objects-filter-options.h | 1 +\n> > list-objects-filter.c | 50 ++++++++++++++++++++++++++\n> > t/t5317-pack-objects-filter-objects.sh | 27 ++++++++++++++\n> > t/t5616-partial-clone.sh | 27 ++++++++++++++\n> > t/t6112-rev-list-filters-objects.sh | 13 +++++++\n> > 7 files changed, 125 insertions(+)\n> >\n> > diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\n> > index 7b273635d..9e351ec2a 100644\n> > --- a/Documentation/rev-list-options.txt\n> > +++ b/Documentation/rev-list-options.txt\n> > @@ -743,6 +743,9 @@ specification contained in .\n> > A debug option to help with future \"partial clone\" development.\n> > This option specifies how missing objects are handled.\n> > +\n> > +The form '--filter=tree:' omits all blobs and trees deeper than\n> > + from the root tree. Currently, only =0 is supported.\n> > ++\n> > The form '--missing=error' requests that rev-list stop with an error if\n> > a missing object is encountered. This is the default action.\n> > +\n> > diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\n> > index c0e2bd6a0..a28382940 100644\n> > --- a/list-objects-filter-options.c\n> > +++ b/list-objects-filter-options.c\n> > @@ -50,6 +50,10 @@ static int gently_parse_list_objects_filter(\n> > return 0;\n> > }\n> >\n> > + } else if (!strcmp(arg, \"tree:0\")) {\n> > + filter_options->choice = LOFC_TREE_NONE;\n> > + return 0;\n> > +\n> > } else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n> > struct object_context oc;\n> > struct object_id sparse_oid;\n> > diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\n> > index 0000a61f8..af64e5c66 100644\n> > --- a/list-objects-filter-options.h\n> > +++ b/list-objects-filter-options.h\n> > @@ -10,6 +10,7 @@ enum list_objects_filter_choice {\n> > LOFC_DISABLED = 0,\n> > LOFC_BLOB_NONE,\n> > LOFC_BLOB_LIMIT,\n> > + LOFC_TREE_NONE,\n> > LOFC_SPARSE_OID,\n> > LOFC_SPARSE_PATH,\n> > LOFC__COUNT /* must be last */\n> > diff --git a/list-objects-filter.c b/list-objects-filter.c\n> > index a0ba78b20..8e3caf5bf 100644\n> > --- a/list-objects-filter.c\n> > +++ b/list-objects-filter.c\n> > @@ -80,6 +80,55 @@ static void *filter_blobs_none__init(\n> > return d;\n> > }\n> >\n> > +/*\n> > + * A filter for list-objects to omit ALL trees and blobs from the traversal.\n> > + * Can OPTIONALLY collect a list of the omitted OIDs.\n> > + */\n> > +struct filter_trees_none_data {\n> > + struct oidset *omits;\n> > +};\n> > +\n> > +static enum list_objects_filter_result filter_trees_none(\n> > + enum list_objects_filter_situation filter_situation,\n> > + struct object *obj,\n> > + const char *pathname,\n> > + const char *filename,\n> > + void *filter_data_)\n> > +{\n> > + struct filter_trees_none_data *filter_data = filter_data_;\n> > +\n> > + switch (filter_situation) {\n> > + default:\n> > + die(\"unknown filter_situation\");\n> > + return LOFR_ZERO;\n> > +\n> > + case LOFS_BEGIN_TREE:\n> > + case LOFS_BLOB:\n> > + if (filter_data->omits)\n> > + oidset_insert(filter_data->omits, &obj->oid);\n> > + return LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n> > +\n> > + case LOFS_END_TREE:\n> > + assert(obj->type == OBJ_TREE);\n> > + return LOFR_ZERO;\n> > +\n> > + }\n> > +}\n>\n> There are a couple of options here:\n> [] If really want to omit all trees and blobs (and we DO NOT want\n> the oidset of everything omitted), then we might be able to\n> shortcut the traversal and speed things up.\n>\n> {} add a LOFR_SKIP_TREE bit to list_objects_filter_result\n> {} test this bit process_tree() and avoid the init_tree_desc() and\n> the while loop and some adjacent setup/tear-down code.\n> {} make this filter something like:\n>\n> case LOFS_BEGIN_TREE:\n> if (filter_data->omits) {\n> oidset_insert(filter_data->omits, &obj->oid);\n> return LOFR_MARK_SEEN; /* ... (hard omit) */\n> } else\n> return LOFR_SKIP_TREE;\n> case LOFS_BLOB:\n> if (filter_data->omits) {\n> oidset_insert(filter_data->omits, &obj->oid);\n> return LOFR_MARK_SEEN; /* ... (hard omit) */\n> else\n> assert(...should not happen...);\nI like this - it will considerably reduce the amount of work the\nserver needs to do on a partial clone. I'd prefer to do this in a\nfollow-up patchset, so I added a NEEDSWORK in the commit that adds\nproper handling for filtered tree objects. I want to make sure the\nunit tests are thorough when I apply your suggestion, and this\npatchset is already a bit more complex than I was expecting.\n\n>\n> [] Later, if we choose to actually support a depth>0, we'll probably\n> want a different filter function to conditionally include/exclude\n> blobs, include shallow tree[node]s, and do some of the provisional-\n> omit logic on deep tree[nodes] (in case a tree appears at multiple\n> places/depths in the history). But that can wait.\n>\n> Jeff\n>\n>\n> > +\n> > +static void* filter_trees_none__init(\n> > + struct oidset *omitted,\n> > + struct list_objects_filter_options *filter_options,\n> > + filter_object_fn *filter_fn,\n> > + filter_free_fn *filter_free_fn)\n> > +{\n> > + struct filter_trees_none_data *d = xcalloc(1, sizeof(*d));\n> > + d->omits = omitted;\n> > +\n> > + *filter_fn = filter_trees_none;\n> > + *filter_free_fn = free;\n> > + return d;\n> > +}\n> > +\n> > /*\n> > * A filter for list-objects to omit large blobs.\n> > * And to OPTIONALLY collect a list of the omitted OIDs.\n> > @@ -374,6 +423,7 @@ static filter_init_fn s_filters[] = {\n> > NULL,\n> > filter_blobs_none__init,\n> > filter_blobs_limit__init,\n> > + filter_trees_none__init,\n> > filter_sparse_oid__init,\n> > filter_sparse_path__init,\n> > };\n> > diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\n> > index 5e35f33bf..65f2cf446 100755\n> > --- a/t/t5317-pack-objects-filter-objects.sh\n> > +++ b/t/t5317-pack-objects-filter-objects.sh\n> > @@ -72,6 +72,33 @@ test_expect_success 'get an error for missing tree object' '\n> > grep -q \"bad tree object\" bad_tree\n> > '\n> >\n> > +test_expect_success 'setup for tests of tree:0' '\n> > + mkdir r1/subtree &&\n> > + echo \"This is a file in a subtree\" > r1/subtree/file &&\n> > + git -C r1 add subtree/file &&\n> > + git -C r1 commit -m subtree\n> > +'\n> > +\n> > +test_expect_success 'verify tree:0 packfile has no blobs or trees' '\n> > + git -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n> > + HEAD\n> > + EOF\n> > + git -C r1 index-pack ../commitsonly.pack &&\n> > + git -C r1 verify-pack -v ../commitsonly.pack >objs &&\n> > + ! grep -E \"tree|blob\" objs\n> > +'\n> > +\n> > +test_expect_success 'grab tree directly when using tree:0' '\n> > + # We should get the tree specified directly but not its blobs or subtrees.\n> > + git -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n> > + HEAD:\n> > + EOF\n> > + git -C r1 index-pack ../commitsonly.pack &&\n> > + git -C r1 verify-pack -v ../commitsonly.pack >objs &&\n> > + grep -E \"tree|blob\" objs >trees_and_blobs &&\n> > + test_line_count = 1 trees_and_blobs\n> > +'\n> > +\n> > # Test blob:limit=[kmg] filter.\n> > # We boundary test around the size parameter. The filter is strictly less than\n> > # the value, so size 500 and 1000 should have the same results, but 1001 should\n> > diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\n> > index bbbe7537d..fc4d182c0 100755\n> > --- a/t/t5616-partial-clone.sh\n> > +++ b/t/t5616-partial-clone.sh\n> > @@ -170,6 +170,33 @@ test_expect_success 'partial clone fetches blobs pointed to by refs even if norm\n> > git -C dst fsck\n> > '\n> >\n> > +test_expect_success 'can use tree:0 to filter partial clone' '\n> > + rm -rf dst &&\n> > + git clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n> > + git -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n> > + cat fetched_objects \\\n> > + | awk -f print_1.awk \\\n> > + | xargs -n1 git -C dst cat-file -t >fetched_types &&\n> > + sort fetched_types -u >unique_types.observed &&\n> > + echo commit > unique_types.expected &&\n> > + test_cmp unique_types.observed unique_types.expected\n> > +'\n> > +\n> > +test_expect_success 'show missing tree objects with --missing=print' '\n> > + git -C dst rev-list master --missing=print --quiet --objects >missing_objs &&\n> > + sed \"s/?//\" missing_objs \\\n> > + | xargs -n1 git -C srv.bare cat-file -t \\\n> > + >missing_types &&\n> > + sort -u missing_types >missing_types.uniq &&\n> > + echo tree >expected &&\n> > + test_cmp missing_types.uniq expected\n> > +'\n> > +\n> > +test_expect_success 'do not complain when a missing tree cannot be parsed' '\n> > + git -C dst rev-list master --missing=print --quiet --objects 2>rev_list_err >&2 &&\n> > + ! grep -q \"Could not read \" rev_list_err\n> > +'\n> > +\n> > . \"$TEST_DIRECTORY\"/lib-httpd.sh\n> > start_httpd\n> >\n> > diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\n> > index 0a37dd5f9..6ccffddbc 100755\n> > --- a/t/t6112-rev-list-filters-objects.sh\n> > +++ b/t/t6112-rev-list-filters-objects.sh\n> > @@ -196,6 +196,19 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n> > test_cmp observed expected\n> > '\n> >\n> > +# Test tree:0 filter.\n> > +\n> > +test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n> > + git -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=tree:0 \\\n> > + | awk -f print_1.awk \\\n> > + | sed s/~// \\\n> > + | xargs -n1 git -C r3 cat-file -t \\\n> > + | sort -u >filtered_types &&\n> > + printf \"blob\\ntree\\n\" > expected &&\n> > + test_cmp filtered_types expected\n> > +'\n> > +\n> > +\n> > # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n> > # This models previously omitted objects that we did not receive.\n> >\n> >\n"},{"id":"355579","messageId":"20180814181841.34446-1-jonathantanmy@google.com","threadId":"49086","inReplyTo":"05e69ef260ca2b28b2444b94850bddf44ca0388a.1534267611.git.matvore@google.com","subject":"Re: [PATCH v4 6/6] list-objects-filter: implement filter tree:0","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-08-14T18:18:41Z","receivedAt":"2018-08-14T18:18:47Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> @@ -743,6 +743,9 @@ specification contained in .\n> \tA debug option to help with future \"partial clone\" development.\n> \tThis option specifies how missing objects are handled.\n> +\n> +The form '--filter=tree:' omits all blobs and trees deeper than\n> + from the root tree. Currently, only =0 is supported.\n> ++\n> The form '--missing=error' requests that rev-list stop with an error if\n> a missing object is encountered. This is the default action.\n> +\n\nThe \"--filter\" documentation should go with the other \"--filter\"\ninformation, not right after --missing.\n\n> +test_expect_success 'setup for tests of tree:0' '\n> +\tmkdir r1/subtree &&\n> +\techo \"This is a file in a subtree\" > r1/subtree/file &&\n> +\tgit -C r1 add subtree/file &&\n> +\tgit -C r1 commit -m subtree\n> +'\n\nStyle: no space after >\n\n> +test_expect_success 'grab tree directly when using tree:0' '\n> +\t# We should get the tree specified directly but not its blobs or subtrees.\n> +\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n> +\tHEAD:\n> +\tEOF\n> +\tgit -C r1 index-pack ../commitsonly.pack &&\n> +\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n> +\tgrep -E \"tree|blob\" objs >trees_and_blobs &&\n> +\ttest_line_count = 1 trees_and_blobs\n> +'\n\nCan we also verify that the SHA-1 in trees_and_blobs is what we\nexpected?\n\n> +test_expect_success 'use fsck before and after manually fetching a missing subtree' '\n> +\t# push new commit so server has a subtree\n> +\tmkdir src/dir &&\n> +\techo \"in dir\" > src/dir/file.txt &&\n\nNo space after >\n\n> +\tgit -C src add dir/file.txt &&\n> +\tgit -C src commit -m \"file in dir\" &&\n> +\tgit -C src push -u srv master &&\n> +\tSUBTREE=$(git -C src rev-parse HEAD:dir) &&\n> +\n> +\trm -rf dst &&\n> +\tgit clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n> +\tgit -C dst fsck &&\n> +\tgit -C dst cat-file -p $SUBTREE >tree_contents 2>err &&\n> +\tgit -C dst fsck\n> +'\n\nIf you don't need to redirect to err, don't do so.\n\nBefore the cat-file, also verify that the tree is missing, most likely\nthrough a \"git rev-list\" with \"--missing=print\".\n\nAnd I would grep on the tree_contents to ensure that the filename\n(\"file.txt\") is there, so that we know that we got the correct tree.\n\n> +test_expect_success 'can use tree:0 to filter partial clone' '\n> +\trm -rf dst &&\n> +\tgit clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n> +\tgit -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n> +\tcat fetched_objects \\\n> +\t\t| awk -f print_1.awk \\\n> +\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n> +\tsort fetched_types -u >unique_types.observed &&\n> +\techo commit > unique_types.expected &&\n> +\ttest_cmp unique_types.observed unique_types.expected\n> +'\n> +\n> +test_expect_success 'auto-fetching of trees with --missing=error' '\n> +\tgit -C dst rev-list master --missing=error --objects >fetched_objects &&\n> +\tcat fetched_objects \\\n> +\t\t| awk -f print_1.awk \\\n> +\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n> +\tsort fetched_types -u >unique_types.observed &&\n> +\tprintf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n> +\ttest_cmp unique_types.observed unique_types.expected\n> +'\n\nThese two tests seem redundant with the 'use fsck before and after\nmanually fetching a missing subtree' test (after the latter is\nappropriately renamed). I think we only need to test this sequence once,\nwhich can be placed in one or spread over multiple tests:\n\n 1. partial clone with --filter=tree:0\n 2. fsck works\n 3. verify that trees are indeed missing\n 4. autofetch a tree\n 5. fsck still works\n"},{"id":"355606","messageId":"CAMfpvhKr4jmjZ3NUmxMyTM7KJxQX30UYXKf_rJ30A4C=P3tB=g@mail.gmail.com","threadId":"49086","inReplyTo":"20180814181841.34446-1-jonathantanmy@google.com","subject":"Re: [PATCH v4 6/6] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-14T20:00:33Z","receivedAt":"2018-08-14T20:00:49Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Tue, Aug 14, 2018 at 11:18 AM Jonathan Tan wrote:\n>\n> > @@ -743,6 +743,9 @@ specification contained in .\n> > A debug option to help with future \"partial clone\" development.\n> > This option specifies how missing objects are handled.\n> > +\n> > +The form '--filter=tree:' omits all blobs and trees deeper than\n> > + from the root tree. Currently, only =0 is supported.\n> > ++\n> > The form '--missing=error' requests that rev-list stop with an error if\n> > a missing object is encountered. This is the default action.\n> > +\n>\n> The \"--filter\" documentation should go with the other \"--filter\"\n> information, not right after --missing.\nFixed. My problem was that I didn't know what the + meant - I guess it\nmeans that the paragraph before and after are in the same section?\n\n>\n> > +test_expect_success 'setup for tests of tree:0' '\n> > + mkdir r1/subtree &&\n> > + echo \"This is a file in a subtree\" > r1/subtree/file &&\n> > + git -C r1 add subtree/file &&\n> > + git -C r1 commit -m subtree\n> > +'\n>\n> Style: no space after >\nFixed.\n\n>\n> > +test_expect_success 'grab tree directly when using tree:0' '\n> > + # We should get the tree specified directly but not its blobs or subtrees.\n> > + git -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n> > + HEAD:\n> > + EOF\n> > + git -C r1 index-pack ../commitsonly.pack &&\n> > + git -C r1 verify-pack -v ../commitsonly.pack >objs &&\n> > + grep -E \"tree|blob\" objs >trees_and_blobs &&\n> > + test_line_count = 1 trees_and_blobs\n> > +'\n>\n> Can we also verify that the SHA-1 in trees_and_blobs is what we\n> expected?\nDone - Now I'm comparing to the output of `git rev-parse HEAD:` and I\ndon't need the separate line count check either.\n>\n> > +test_expect_success 'use fsck before and after manually fetching a missing subtree' '\n> > + # push new commit so server has a subtree\n> > + mkdir src/dir &&\n> > + echo \"in dir\" > src/dir/file.txt &&\n>\n> No space after >\nFixed.\n\n>\n> > + git -C src add dir/file.txt &&\n> > + git -C src commit -m \"file in dir\" &&\n> > + git -C src push -u srv master &&\n> > + SUBTREE=$(git -C src rev-parse HEAD:dir) &&\n> > +\n> > + rm -rf dst &&\n> > + git clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n> > + git -C dst fsck &&\n> > + git -C dst cat-file -p $SUBTREE >tree_contents 2>err &&\n> > + git -C dst fsck\n> > +'\n>\n> If you don't need to redirect to err, don't do so.\n>\n> Before the cat-file, also verify that the tree is missing, most likely\n> through a \"git rev-list\" with \"--missing=print\".\nThat won't work though - the subtree's hash is not known because its\nparent tree is not there. I've merged the three tests in this file,\nand as a result am now using the check which makes sure the object\ntypes are only \"commit\"\n\n>\n> And I would grep on the tree_contents to ensure that the filename\n> (\"file.txt\") is there, so that we know that we got the correct tree.\nDone.\n\n>\n> > +test_expect_success 'can use tree:0 to filter partial clone' '\n> > + rm -rf dst &&\n> > + git clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n> > + git -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n> > + cat fetched_objects \\\n> > + | awk -f print_1.awk \\\n> > + | xargs -n1 git -C dst cat-file -t >fetched_types &&\n> > + sort fetched_types -u >unique_types.observed &&\n> > + echo commit > unique_types.expected &&\n> > + test_cmp unique_types.observed unique_types.expected\n> > +'\n> > +\n> > +test_expect_success 'auto-fetching of trees with --missing=error' '\n> > + git -C dst rev-list master --missing=error --objects >fetched_objects &&\n> > + cat fetched_objects \\\n> > + | awk -f print_1.awk \\\n> > + | xargs -n1 git -C dst cat-file -t >fetched_types &&\n> > + sort fetched_types -u >unique_types.observed &&\n> > + printf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n> > + test_cmp unique_types.observed unique_types.expected\n> > +'\n>\n> These two tests seem redundant with the 'use fsck before and after\n> manually fetching a missing subtree' test (after the latter is\n> appropriately renamed). I think we only need to test this sequence once,\n> which can be placed in one or spread over multiple tests:\n>\n> 1. partial clone with --filter=tree:0\n> 2. fsck works\n> 3. verify that trees are indeed missing\n> 4. autofetch a tree\n> 5. fsck still works\nDone - that's much nicer. Thanks!\n\nHere is an interdiff from v4 of the patch:\n\ndiff --git a/Documentation/rev-list-options.txt\nb/Documentation/rev-list-options.txt\nindex 9e351ec2a..0b5f77ad3 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -731,6 +731,9 @@ the requested refs.\n +\n The form '--filter=sparse:path=' similarly uses a sparse-checkout\n specification contained in .\n++\n+The form '--filter=tree:' omits all blobs and trees deeper than\n+ from the root tree. Currently, only =0 is supported.\n\n --no-filter::\n Turn off any previous `--filter=` argument.\n@@ -743,9 +746,6 @@ specification contained in .\n A debug option to help with future \"partial clone\" development.\n This option specifies how missing objects are handled.\n +\n-The form '--filter=tree:' omits all blobs and trees deeper than\n- from the root tree. Currently, only =0 is supported.\n-+\n The form '--missing=error' requests that rev-list stop with an error if\n a missing object is encountered. This is the default action.\n +\ndiff --git a/t/t5317-pack-objects-filter-objects.sh\nb/t/t5317-pack-objects-filter-objects.sh\nindex 65f2cf446..fe7c13a03 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -74,7 +74,7 @@ test_expect_success 'get an error for missing tree object' '\n\n test_expect_success 'setup for tests of tree:0' '\n mkdir r1/subtree &&\n- echo \"This is a file in a subtree\" > r1/subtree/file &&\n+ echo \"This is a file in a subtree\" >r1/subtree/file &&\n git -C r1 add subtree/file &&\n git -C r1 commit -m subtree\n '\n@@ -95,8 +95,10 @@ test_expect_success 'grab tree directly when using tree:0' '\n EOF\n git -C r1 index-pack ../commitsonly.pack &&\n git -C r1 verify-pack -v ../commitsonly.pack >objs &&\n- grep -E \"tree|blob\" objs >trees_and_blobs &&\n- test_line_count = 1 trees_and_blobs\n+ grep -E \"tree|blob\" objs \\\n+ | awk -f print_1.awk >trees_and_blobs &&\n+ git -C r1 rev-parse HEAD: >expected &&\n+ test_cmp trees_and_blobs expected\n '\n\n # Test blob:limit=[kmg] filter.\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex d2859aba1..c3186d934 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -157,7 +157,7 @@ test_expect_success 'partial clone with\ntransfer.fsckobjects=1 uses index-pack -\n test_expect_success 'use fsck before and after manually fetching a\nmissing subtree' '\n # push new commit so server has a subtree\n mkdir src/dir &&\n- echo \"in dir\" > src/dir/file.txt &&\n+ echo \"in dir\" >src/dir/file.txt &&\n git -C src add dir/file.txt &&\n git -C src commit -m \"file in dir\" &&\n git -C src push -u srv master &&\n@@ -166,8 +166,32 @@ test_expect_success 'use fsck before and after\nmanually fetching a missing subtr\n rm -rf dst &&\n git clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n git -C dst fsck &&\n- git -C dst cat-file -p $SUBTREE >tree_contents 2>err &&\n- git -C dst fsck\n+\n+ # Make sure we only have commits, and all trees and blobs are missing.\n+ git -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n+ cat fetched_objects \\\n+ | awk -f print_1.awk \\\n+ | xargs -n1 git -C dst cat-file -t >fetched_types &&\n+ sort fetched_types -u >unique_types.observed &&\n+ echo commit >unique_types.expected &&\n+ test_cmp unique_types.observed unique_types.expected &&\n+\n+ # Auto-fetch a tree with cat-file.\n+ git -C dst cat-file -p $SUBTREE >tree_contents &&\n+ grep file.txt tree_contents &&\n+\n+ # fsck still works after an auto-fetch of a tree.\n+ git -C dst fsck &&\n+\n+ # Auto-fetch all remaining trees and blobs with --missing=error\n+ git -C dst rev-list master --missing=error --objects >fetched_objects &&\n+ test_line_count = 70 fetched_objects &&\n+ cat fetched_objects \\\n+ | awk -f print_1.awk \\\n+ | xargs -n1 git -C dst cat-file -t >fetched_types &&\n+ sort fetched_types -u >unique_types.observed &&\n+ printf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n+ test_cmp unique_types.observed unique_types.expected\n '\n\n test_expect_success 'partial clone fetches blobs pointed to by refs\neven if normally filtered out' '\n@@ -186,28 +210,6 @@ test_expect_success 'partial clone fetches blobs\npointed to by refs even if norm\n git -C dst fsck\n '\n\n-test_expect_success 'can use tree:0 to filter partial clone' '\n- rm -rf dst &&\n- git clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n- git -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n- cat fetched_objects \\\n- | awk -f print_1.awk \\\n- | xargs -n1 git -C dst cat-file -t >fetched_types &&\n- sort fetched_types -u >unique_types.observed &&\n- echo commit > unique_types.expected &&\n- test_cmp unique_types.observed unique_types.expected\n-'\n-\n-test_expect_success 'auto-fetching of trees with --missing=error' '\n- git -C dst rev-list master --missing=error --objects >fetched_objects &&\n- cat fetched_objects \\\n- | awk -f print_1.awk \\\n- | xargs -n1 git -C dst cat-file -t >fetched_types &&\n- sort fetched_types -u >unique_types.observed &&\n- printf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n- test_cmp unique_types.observed unique_types.expected\n-'\n-\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n"},{"id":"355607","messageId":"20180814200154.GF28452@sigill.intra.peff.net","threadId":"49086","inReplyTo":"05e69ef260ca2b28b2444b94850bddf44ca0388a.1534267611.git.matvore@google.com","subject":"Re: [PATCH v4 6/6] list-objects-filter: implement filter tree:0","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-08-14T20:01:55Z","receivedAt":"2018-08-14T20:01:58Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 14, 2018 at 10:28:13AM -0700, Matthew DeVore wrote:\n\n> The name \"tree:0\" allows later filtering based on depth, i.e. \"tree:1\"\n> would filter out all but the root tree and blobs. In order to avoid\n> confusion between 0 and capital O, the documentation was worded in a\n> somewhat round-about way that also hints at this future improvement to\n> the feature.\n\nI'm OK with this as a name, since we're explicitly not supporting deeper\ndepths. But I'd note that \"depth\" is actually a tricky characteristic,\nas it's not a property of the object itself, but rather who refers to\nit. So:\n\n - it's expensive to compute, because you have to actually walk all of\n the possible commits and trees that could refer to it. This\n prohibits a lot of other optimizations like reachability bitmaps\n (though with some complexity you could cache the depths, too).\n\n - you have to define it as something like \"the minimum depth at which\n this object is found\", since there may be multiple depths\n\nI think you can read that second definition between the lines of:\n\n> +The form '--filter=tree:' omits all blobs and trees deeper than\n> + from the root tree. Currently, only =0 is supported.\n\nBut I wonder if we should be more precise. It doesn't matter now, but it\nmay help set expectations if the feature does come later.\n\n-Peff\n"},{"id":"355612","messageId":"20180814201920.56534-1-jonathantanmy@google.com","threadId":"49086","inReplyTo":"CAMfpvhKr4jmjZ3NUmxMyTM7KJxQX30UYXKf_rJ30A4C=P3tB=g@mail.gmail.com","subject":"Re: [PATCH v4 6/6] list-objects-filter: implement filter tree:0","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-08-14T20:19:20Z","receivedAt":"2018-08-14T20:19:26Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> - grep -E \"tree|blob\" objs >trees_and_blobs &&\n> - test_line_count = 1 trees_and_blobs\n> + grep -E \"tree|blob\" objs \\\n> + | awk -f print_1.awk >trees_and_blobs &&\n> + git -C r1 rev-parse HEAD: >expected &&\n> + test_cmp trees_and_blobs expected\n\nIndent \"| awk\" (and similar lines in this patch) - although I guess it\nis likely that you actually have it indented, and your e-mail client\nmodified the whitespace so that it looks like there is no indent.\n\nOther than that, this interdiff looks good to me. Thanks.\n"},{"id":"355616","messageId":"xmqqa7pozmzv.fsf@gitster-ct.c.googlers.com","threadId":"49086","inReplyTo":"20180814201920.56534-1-jonathantanmy@google.com","subject":"Re: [PATCH v4 6/6] list-objects-filter: implement filter tree:0","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-08-14T20:55:48Z","receivedAt":"2018-08-14T20:55:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n>> - grep -E \"tree|blob\" objs >trees_and_blobs &&\n>> - test_line_count = 1 trees_and_blobs\n>> + grep -E \"tree|blob\" objs \\\n>> + | awk -f print_1.awk >trees_and_blobs &&\n>> + git -C r1 rev-parse HEAD: >expected &&\n>> + test_cmp trees_and_blobs expected\n>\n> Indent \"| awk\" (and similar lines in this patch) - although I guess it\n> is likely that you actually have it indented, and your e-mail client\n> modified the whitespace so that it looks like there is no indent.\n\nNo, wrap lines like this\n\n\tcommand1 arg1 arg2 |\n\tcommand2 arg1 arg2 &&\n\nThat way, you do not need backslash to continue line.\n\nAlso think twice when you are seeing yourself piping output from\n\"grep\" to more powerful tools like \"perl\", \"awk\" or \"sed\". Often\nyou can lose the upstream \"grep\".\n"},{"id":"355639","messageId":"CAMfpvh+2SeOynnzH_keOPE9g2PrMgfzrjOMHWxQxkuOf+T3Y7A@mail.gmail.com","threadId":"49086","inReplyTo":"20180814180626.31788-1-jonathantanmy@google.com","subject":"Re: [PATCH v4 4/6] rev-list: handle missing tree objects properly","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-14T22:43:25Z","receivedAt":"2018-08-14T22:43:40Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Tue, Aug 14, 2018 at 11:06 AM Jonathan Tan wrote:\n>\n> > Previously, we assumed only blob objects could be missing. This patch\n> > makes rev-list handle missing trees like missing blobs. A missing tree\n> > will cause an error if --missing indicates an error should be caused,\n> > and the hash is printed even if the tree is missing.\n>\n> The last sentence is difficult to understand - probably better to say\n> that all --missing= arguments and --exclude-promisor-objects work for\n> missing trees like they currently do for blobs (and do not fixate on\n> just --missing=error). And also demonstrate this in tests, like in\n> t6612.\nFixed the commit message. And for the tests, in t0410 I changed the\n--missing=allow-any to --missing-allow-promisor, and in t6112 I added\n--missing=allow-any and --missing=print test cases.\n\n>\n> > In list-objects.c we no longer print a message to stderr if a tree\n> > object is missing (quiet_on_missing is always true). I couldn't find\n> > any place where this would matter, or where the caller of\n> > traverse_commit_list would need to be fixed to show the error. However,\n> > in the future it would be trivial to make the caller show the message if\n> > we needed to.\n> >\n> > This is not tested very thoroughly, since we cannot create promisor\n> > objects in tests without using an actual partial clone. t0410 has a\n> > promise_and_delete utility function, but the is_promisor_object function\n> > does not return 1 for objects deleted in this way. More tests will will\n> > come in a patch that implements a filter that can be used with git\n> > clone.\n>\n> These two paragraphs are no longer applicable, I think.\nSorry about that. Removed.\n\n>\n> > @@ -373,6 +375,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n> > init_revisions(&revs, prefix);\n> > revs.abbrev = DEFAULT_ABBREV;\n> > revs.commit_format = CMIT_FMT_UNSPECIFIED;\n> > + revs.do_not_die_on_missing_tree = 1;\n>\n> Is this correct? I would have expected this to be set only if --missing\n> was set.\nIf --missing is not set, then we want to fetch missing objects\nautomatically, and then die if we fail to do that, which is what\nhappens for blobs. So we don't want to die in list-objects.c. If we\nfail to fetch, then we will die on line 213 in rev-list.c.\n\n>\n> > - process_tree_contents(ctx, tree, base);\n> > + /*\n> > + * NEEDSWORK: we should not have to process this tree's contents if the\n> > + * filter wants to exclude all its contents AND the filter doesn't need\n> > + * to collect the omitted OIDs. We should add a LOFR_SKIP_TREE bit which\n> > + * allows skipping all children.\n> > + */\n> > + if (parsed)\n> > + process_tree_contents(ctx, tree, base);\n>\n> I agree with Jeff Hostetler in [1] that a LOFR_SKIP_TREE bit is\n> desirable, but I don't think that this patch is the right place to\n> introduce this NEEDSWORK. For me, this patch is about skipping iterating\n> over the contents of a tree because the tree does not exist; this\n> NEEDSWORK is about skipping iterating over the contents of a tree\n> because we don't want its contents, and it is quite confusing to\n> conflate the two.\nI've removed this.\n\n>\n> [1] https://public-inbox.org/git/d751d56b-84bb-a03d-5f2a-7dbaf8d947cc@jeffhostetler.com/\n>\n> > @@ -124,6 +124,7 @@ struct rev_info {\n> > first_parent_only:1,\n> > line_level_traverse:1,\n> > tree_blobs_in_commit_order:1,\n> > + do_not_die_on_missing_tree:1,\n>\n> I know that the other flags don't have documentation, but I think it's\n> worth documenting this one because it is rather complicated. I have\n> provided a sample one in my earlier review - feel free to use that or\n> come up with your own.\nAdded your wording to revision.h without major change.\n\n>\n> > diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\n> > index 4984ca583..74e3c5767 100755\n> > --- a/t/t0410-partial-clone.sh\n> > +++ b/t/t0410-partial-clone.sh\n> > @@ -186,6 +186,72 @@ test_expect_success 'rev-list stops traversal at missing and promised commit' '\n> > ! grep $FOO out\n> > '\n> >\n> > +test_expect_success 'show missing tree objects with --missing=print' '\n> > + rm -rf repo &&\n> > + test_create_repo repo &&\n> > + test_commit -C repo foo &&\n> > + test_commit -C repo bar &&\n> > + test_commit -C repo baz &&\n> > +\n> > + TREE=$(git -C repo rev-parse bar^{tree}) &&\n> > +\n> > + promise_and_delete $TREE &&\n> > +\n> > + git -C repo config core.repositoryformatversion 1 &&\n> > + git -C repo config extensions.partialclone \"arbitrary string\" &&\n> > + git -C repo rev-list --quiet --missing=print --objects HEAD >missing_objs 2>rev_list_err &&\n> > + echo \"?$TREE\" >expected &&\n> > + test_cmp expected missing_objs &&\n> > +\n> > + # do not complain when a missing tree cannot be parsed\n> > + ! grep -q \"Could not read \" rev_list_err\n> > +'\n>\n> I think that the --exclude-promisor-tests can go in t0410 as you have\n> done, but the --missing tests (except for --missing=allow-promisor)\n> should go in t6112. (And like the existing --missing tests, they should\n> be done without setting extensions.partialclone.)\nDone.\n\n>\n> As for --missing=allow-promisor, I don't see them being tested anywhere\n> :-( so feel free to make a suggestion. I would put them in t6112 for\n> easy comparison with the other --missing tests.\nKept my allow-promisor test in t0410 since it requires partial clone\nto be turned on in the config, and because it is pretty similar to\n--exclude-promisor-objects.\n"},{"id":"355640","messageId":"20180814225633.90251-1-jonathantanmy@google.com","threadId":"49086","inReplyTo":"CAMfpvh+2SeOynnzH_keOPE9g2PrMgfzrjOMHWxQxkuOf+T3Y7A@mail.gmail.com","subject":"Re: [PATCH v4 4/6] rev-list: handle missing tree objects properly","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-08-14T22:56:33Z","receivedAt":"2018-08-14T22:56:38Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > > @@ -373,6 +375,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n> > > init_revisions(&revs, prefix);\n> > > revs.abbrev = DEFAULT_ABBREV;\n> > > revs.commit_format = CMIT_FMT_UNSPECIFIED;\n> > > + revs.do_not_die_on_missing_tree = 1;\n> >\n> > Is this correct? I would have expected this to be set only if --missing\n> > was set.\n> If --missing is not set, then we want to fetch missing objects\n> automatically, and then die if we fail to do that, which is what\n> happens for blobs.\n\nThis is true, and should already be handled. Pay attention to when\nfetch_if_missing is set in builtin/rev-list.c.\n\ndo_not_die_on_missing_tree should probably be set to 1 whenever\nfetch_if_missing is set to 0, I think.\n\n(I acknowledge that the usage of this global variable is confusing, but\nI couldn't think of a better way to implement this when I did. Perhaps\nwhen the object store refactoring is done, this can be a store-specific\nsetting instead of a global variable.)\n\n> So we don't want to die in list-objects.c. If we\n> fail to fetch, then we will die on line 213 in rev-list.c.\n\nWhy don't we want to die in list-objects.c? When --missing=error is\npassed, fetch_if_missing retains its default value of 1, so\nparse_tree_gently() will attempt to fetch it - and if it fails, I think\nit's appropriate to die in list-objects.c (and this should be the\ncurrent behavior). On other values, e.g. --missing=allow-any, there is\nno autofetch (since fetch_if_missing is 0), so it is correct not to die\nin list-objects.c.\n\n> > As for --missing=allow-promisor, I don't see them being tested anywhere\n> > :-( so feel free to make a suggestion. I would put them in t6112 for\n> > easy comparison with the other --missing tests.\n> Kept my allow-promisor test in t0410 since it requires partial clone\n> to be turned on in the config, and because it is pretty similar to\n> --exclude-promisor-objects.\n\nOK, sounds good to me.\n"},{"id":"355645","messageId":"20180814231425.94903-1-jonathantanmy@google.com","threadId":"49086","inReplyTo":"20180814225633.90251-1-jonathantanmy@google.com","subject":"Re: [PATCH v4 4/6] rev-list: handle missing tree objects properly","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-08-14T23:14:25Z","receivedAt":"2018-08-14T23:14:31Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > So we don't want to die in list-objects.c. If we\n> > fail to fetch, then we will die on line 213 in rev-list.c.\n> \n> Why don't we want to die in list-objects.c? When --missing=error is\n> passed, fetch_if_missing retains its default value of 1, so\n> parse_tree_gently() will attempt to fetch it - and if it fails, I think\n> it's appropriate to die in list-objects.c (and this should be the\n> current behavior). On other values, e.g. --missing=allow-any, there is\n> no autofetch (since fetch_if_missing is 0), so it is correct not to die\n> in list-objects.c.\n\nAfter some in-office discussion, I should have checked line 213 in\nbuiltin/rev-list.c more thorougly. Indeed it is OK not to die in\nlist-objects.c here, since builtin/rev-list.c already knows how to\nhandle missing objects in the --missing=error circumstance.\n"},{"id":"355650","messageId":"CAMfpvh+H5UeHmCqeVNZNoOe_-vgH3_3aTuO+c=imgzaVkC6N2g@mail.gmail.com","threadId":"49086","inReplyTo":"xmqqa7pozmzv.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 6/6] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-14T23:30:37Z","receivedAt":"2018-08-14T23:30:51Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Tue, Aug 14, 2018 at 1:55 PM Junio C Hamano wrote:\n>\n> Jonathan Tan writes:\n>\n> >> - grep -E \"tree|blob\" objs >trees_and_blobs &&\n> >> - test_line_count = 1 trees_and_blobs\n> >> + grep -E \"tree|blob\" objs \\\n> >> + | awk -f print_1.awk >trees_and_blobs &&\n> >> + git -C r1 rev-parse HEAD: >expected &&\n> >> + test_cmp trees_and_blobs expected\n> >\n> > Indent \"| awk\" (and similar lines in this patch) - although I guess it\n> > is likely that you actually have it indented, and your e-mail client\n> > modified the whitespace so that it looks like there is no indent.\n>\n> No, wrap lines like this\n>\n> command1 arg1 arg2 |\n> command2 arg1 arg2 &&\n>\n> That way, you do not need backslash to continue line.\n>\n> Also think twice when you are seeing yourself piping output from\n> \"grep\" to more powerful tools like \"perl\", \"awk\" or \"sed\". Often\n> you can lose the upstream \"grep\".\n\nThank you. I changed it to this:\n awk -e \"/tree|blob/{print \\$1}\" objs >trees_and_blobs\n\nAbout the line wrapping strategy, the files I've edited all did it\nwith the \\ and the | at the start of subsequent lines - so I made my\ncode match that style. Otherwise I would have liked to use the style\nyou suggest...\n"},{"id":"355651","messageId":"CAMfpvhJM2hUXTfTgY9jA4aB+bgx44xbCktCacpy_yWacyjk00w@mail.gmail.com","threadId":"49086","inReplyTo":"20180814200154.GF28452@sigill.intra.peff.net","subject":"Re: [PATCH v4 6/6] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-14T23:55:34Z","receivedAt":"2018-08-14T23:55:49Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Tue, Aug 14, 2018 at 1:01 PM Jeff King wrote:\n>\n> On Tue, Aug 14, 2018 at 10:28:13AM -0700, Matthew DeVore wrote:\n>\n> > The name \"tree:0\" allows later filtering based on depth, i.e. \"tree:1\"\n> > would filter out all but the root tree and blobs. In order to avoid\n> > confusion between 0 and capital O, the documentation was worded in a\n> > somewhat round-about way that also hints at this future improvement to\n> > the feature.\n>\n> I'm OK with this as a name, since we're explicitly not supporting deeper\n> depths. But I'd note that \"depth\" is actually a tricky characteristic,\n> as it's not a property of the object itself, but rather who refers to\n> it. So:\n>\n> - it's expensive to compute, because you have to actually walk all of\n> the possible commits and trees that could refer to it. This\n> prohibits a lot of other optimizations like reachability bitmaps\n> (though with some complexity you could cache the depths, too).\nI think what the user likely wants is to use the minimum depth based\non the commits in the traversal, not every commit in the repo - is\nthis what you mean?\n\n>\n> - you have to define it as something like \"the minimum depth at which\n> this object is found\", since there may be multiple depths\n>\n> I think you can read that second definition between the lines of:\n>\n> > +The form '--filter=tree:' omits all blobs and trees deeper than\n> > + from the root tree. Currently, only =0 is supported.\n>\n> But I wonder if we should be more precise. It doesn't matter now, but it\n> may help set expectations if the feature does come later.\n>\nMakes sense. I changed it like this -\n\ndiff --git a/Documentation/rev-list-options.txt\nb/Documentation/rev-list-options.txt\nindex 0b5f77ad3..5f1672913 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -732,8 +732,10 @@ the requested refs.\n The form '--filter=sparse:path=' similarly uses a sparse-checkout\n specification contained in .\n +\n-The form '--filter=tree:' omits all blobs and trees deeper than\n- from the root tree. Currently, only =0 is supported.\n+The form '--filter=tree:' omits all blobs and trees whose depth\n+from the root tree is >= (minimum depth if an object is located\n+at multiple depths in the commits traversed). Currently, only =0\n+is supported, which omits all blobs and trees.\n\n --no-filter::\n Turn off any previous `--filter=` argument.\n\n\n> -Peff\n"},{"id":"355652","messageId":"cover.1534291033.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"[PATCH v5 0/6] filter: support for excluding all trees and blobs","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-15T00:22:49Z","receivedAt":"2018-08-15T00:23:04Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Please take a look. I believe I have applied or responded to all suggestions\nsince the last iteration.\n\nMatthew DeVore (6):\n list-objects: store common func args in struct\n list-objects: refactor to process_tree_contents\n list-objects: always parse trees gently\n rev-list: handle missing tree objects properly\n revision: mark non-user-given objects instead\n list-objects-filter: implement filter tree:0\n\n Documentation/rev-list-options.txt | 5 +\n builtin/rev-list.c | 11 +-\n list-objects-filter-options.c | 4 +\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 50 ++++++\n list-objects.c | 232 +++++++++++++------------\n revision.c | 1 -\n revision.h | 25 ++-\n t/t0410-partial-clone.sh | 45 +++++\n t/t5317-pack-objects-filter-objects.sh | 41 +++++\n t/t5616-partial-clone.sh | 38 ++++\n t/t6112-rev-list-filters-objects.sh | 29 ++++\n 12 files changed, 364 insertions(+), 118 deletions(-)\n\n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355653","messageId":"c54a2818125739b7dde40aeee2cf1d72e057e963.1534291033.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534291033.git.matvore@google.com","subject":"[PATCH v5 1/6] list-objects: store common func args in struct","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-15T00:22:50Z","receivedAt":"2018-08-15T00:23:07Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will make utility functions easier to create, as done by the next\npatch.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 158 +++++++++++++++++++++++--------------------------\n 1 file changed, 74 insertions(+), 84 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex c99c47ac1..584518a3f 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -12,20 +12,25 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n \n-static void process_blob(struct rev_info *revs,\n+struct traversal_context {\n+\tstruct rev_info *revs;\n+\tshow_object_fn show_object;\n+\tshow_commit_fn show_commit;\n+\tvoid *show_data;\n+\tfilter_object_fn filter_fn;\n+\tvoid *filter_data;\n+};\n+\n+static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \n-\tif (!revs->blob_objects)\n+\tif (!ctx->revs->blob_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad blob object\");\n@@ -41,21 +46,21 @@ static void process_blob(struct rev_info *revs,\n \t * may cause the actual filter to report an incomplete list\n \t * of missing objects.\n \t */\n-\tif (revs->exclude_promisor_objects &&\n+\tif (ctx->revs->exclude_promisor_objects &&\n \t !has_object_file(&obj->oid) &&\n \t is_promisor_object(&obj->oid))\n \t\treturn;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BLOB, obj,\n-\t\t\t path->buf, &path->buf[pathlen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n+\t\t\t\t path->buf, &path->buf[pathlen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, path->buf, cb_data);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -81,26 +86,21 @@ static void process_blob(struct rev_info *revs,\n * the link, and how to do it. Whether it necessarily makes\n * any sense what-so-ever to ever do that is another issue.\n */\n-static void process_gitlink(struct rev_info *revs,\n+static void process_gitlink(struct traversal_context *ctx,\n \t\t\t const unsigned char *sha1,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data)\n+\t\t\t const char *name)\n {\n \t/* Nothing to do */\n }\n \n-static void process_tree(struct rev_info *revs,\n+static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &tree->object;\n+\tstruct rev_info *revs = ctx->revs;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n@@ -133,14 +133,14 @@ static void process_tree(struct rev_info *revs,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BEGIN_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, base->buf, cb_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -157,29 +157,25 @@ static void process_tree(struct rev_info *revs,\n \t\t}\n \n \t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(revs,\n+\t\t\tprocess_tree(ctx,\n \t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(revs, entry.oid->hash,\n-\t\t\t\t\tshow, base, entry.path,\n-\t\t\t\t\tcb_data);\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n \t\telse\n-\t\t\tprocess_blob(revs,\n+\t\t\tprocess_blob(ctx,\n \t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t}\n \n-\tif (!(obj->flags & USER_GIVEN) && filter_fn) {\n-\t\tr = filter_fn(LOFS_END_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \t\tif (r & LOFR_MARK_SEEN)\n \t\t\tobj->flags |= SEEN;\n \t\tif (r & LOFR_DO_SHOW)\n-\t\t\tshow(obj, base->buf, cb_data);\n+\t\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \t}\n \n \tstrbuf_setlen(base, baselen);\n@@ -242,19 +238,15 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n-static void traverse_trees_and_blobs(struct rev_info *revs,\n-\t\t\t\t struct strbuf *base,\n-\t\t\t\t show_object_fn show_object,\n-\t\t\t\t void *show_data,\n-\t\t\t\t filter_object_fn filter_fn,\n-\t\t\t\t void *filter_data)\n+static void traverse_trees_and_blobs(struct traversal_context *ctx,\n+\t\t\t\t struct strbuf *base)\n {\n \tint i;\n \n \tassert(base->len == 0);\n \n-\tfor (i = 0; i < revs->pending.nr; i++) {\n-\t\tstruct object_array_entry *pending = revs->pending.objects + i;\n+\tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n+\t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n@@ -262,62 +254,49 @@ static void traverse_trees_and_blobs(struct rev_info *revs,\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tshow_object(obj, name, show_data);\n+\t\t\tctx->show_object(obj, name, ctx->show_data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(revs, (struct tree *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(revs, (struct blob *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n \t\t oid_to_hex(&obj->oid), name);\n \t}\n-\tobject_array_clear(&revs->pending);\n+\tobject_array_clear(&ctx->revs->pending);\n }\n \n-static void do_traverse(struct rev_info *revs,\n-\t\t\tshow_commit_fn show_commit,\n-\t\t\tshow_object_fn show_object,\n-\t\t\tvoid *show_data,\n-\t\t\tfilter_object_fn filter_fn,\n-\t\t\tvoid *filter_data)\n+static void do_traverse(struct traversal_context *ctx)\n {\n \tstruct commit *commit;\n \tstruct strbuf csp; /* callee's scratch pad */\n \tstrbuf_init(&csp, PATH_MAX);\n \n-\twhile ((commit = get_revision(revs)) != NULL) {\n+\twhile ((commit = get_revision(ctx->revs)) != NULL) {\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n \t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(revs, get_commit_tree(commit));\n-\t\tshow_commit(commit, show_data);\n+\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tctx->show_commit(commit, ctx->show_data);\n \n-\t\tif (revs->tree_blobs_in_commit_order)\n+\t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n \t\t\t * NEEDSWORK: Adding the tree and then flushing it here\n \t\t\t * needs a reallocation for each commit. Can we pass the\n \t\t\t * tree directory without allocation churn?\n \t\t\t */\n-\t\t\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t\t\t show_object, show_data,\n-\t\t\t\t\t\t filter_fn, filter_data);\n+\t\t\ttraverse_trees_and_blobs(ctx, &csp);\n \t}\n-\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t show_object, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\ttraverse_trees_and_blobs(ctx, &csp);\n \tstrbuf_release(&csp);\n }\n \n@@ -326,7 +305,14 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t\t show_object_fn show_object,\n \t\t\t void *show_data)\n {\n-\tdo_traverse(revs, show_commit, show_object, show_data, NULL, NULL);\n+\tstruct traversal_context ctx;\n+\tctx.revs = revs;\n+\tctx.show_commit = show_commit;\n+\tctx.show_object = show_object;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\tctx.filter_data = NULL;\n+\tdo_traverse(&ctx);\n }\n \n void traverse_commit_list_filtered(\n@@ -337,14 +323,18 @@ void traverse_commit_list_filtered(\n \tvoid *show_data,\n \tstruct oidset *omitted)\n {\n-\tfilter_object_fn filter_fn = NULL;\n+\tstruct traversal_context ctx;\n \tfilter_free_fn filter_free_fn = NULL;\n-\tvoid *filter_data = NULL;\n-\n-\tfilter_data = list_objects_filter__init(omitted, filter_options,\n-\t\t\t\t\t\t&filter_fn, &filter_free_fn);\n-\tdo_traverse(revs, show_commit, show_object, show_data,\n-\t\t filter_fn, filter_data);\n-\tif (filter_data && filter_free_fn)\n-\t\tfilter_free_fn(filter_data);\n+\n+\tctx.revs = revs;\n+\tctx.show_object = show_object;\n+\tctx.show_commit = show_commit;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\n+\tctx.filter_data = list_objects_filter__init(omitted, filter_options,\n+\t\t\t\t\t\t &ctx.filter_fn, &filter_free_fn);\n+\tdo_traverse(&ctx);\n+\tif (ctx.filter_data && filter_free_fn)\n+\t\tfilter_free_fn(ctx.filter_data);\n }\n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355654","messageId":"feb7ea58228df4bb777ee96651389000e804f9fa.1534291033.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534291033.git.matvore@google.com","subject":"[PATCH v5 2/6] list-objects: refactor to process_tree_contents","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-15T00:22:51Z","receivedAt":"2018-08-15T00:23:09Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will be used in a follow-up patch to reduce indentation needed when\ninvoking the logic conditionally. i.e. rather than:\n\nif (foo) {\n\twhile (...) {\n\t\t/* this is very indented */\n\t}\n}\n\nwe will have:\n\nif (foo)\n\tprocess_tree_contents(...);\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 68 ++++++++++++++++++++++++++++++--------------------\n 1 file changed, 41 insertions(+), 27 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 584518a3f..ccc529e5e 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -94,6 +94,46 @@ static void process_gitlink(struct traversal_context *ctx,\n \t/* Nothing to do */\n }\n \n+static void process_tree(struct traversal_context *ctx,\n+\t\t\t struct tree *tree,\n+\t\t\t struct strbuf *base,\n+\t\t\t const char *name);\n+\n+static void process_tree_contents(struct traversal_context *ctx,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *base)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tenum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ?\n+\t\tall_entries_interesting : entry_not_interesting;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tif (match != all_entries_interesting) {\n+\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n+\t\t\t\t\t\t &ctx->revs->diffopt.pathspec);\n+\t\t\tif (match == all_entries_not_interesting)\n+\t\t\t\tbreak;\n+\t\t\tif (match == entry_not_interesting)\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (S_ISDIR(entry.mode))\n+\t\t\tprocess_tree(ctx,\n+\t\t\t\t lookup_tree(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t\telse if (S_ISGITLINK(entry.mode))\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash,\n+\t\t\t\t\tbase, entry.path);\n+\t\telse\n+\t\t\tprocess_blob(ctx,\n+\t\t\t\t lookup_blob(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t}\n+}\n+\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n@@ -101,10 +141,6 @@ static void process_tree(struct traversal_context *ctx,\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n-\tstruct tree_desc desc;\n-\tstruct name_entry entry;\n-\tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n-\t\tall_entries_interesting: entry_not_interesting;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \tint gently = revs->ignore_missing_links ||\n@@ -144,29 +180,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tinit_tree_desc(&desc, tree->buffer, tree->size);\n-\n-\twhile (tree_entry(&desc, &entry)) {\n-\t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n-\t\t\t\t\t\t &revs->diffopt.pathspec);\n-\t\t\tif (match == all_entries_not_interesting)\n-\t\t\t\tbreak;\n-\t\t\tif (match == entry_not_interesting)\n-\t\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t}\n+\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355655","messageId":"7b9ffe47c4e91d442078e1ed99b53653f5fed6a3.1534291033.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534291033.git.matvore@google.com","subject":"[PATCH v5 3/6] list-objects: always parse trees gently","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-15T00:22:52Z","receivedAt":"2018-08-15T00:23:12Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"If parsing fails when revs->ignore_missing_links and\nrevs->exclude_promisor_objects are both false, we print the OID anyway\nin the die(\"bad tree object...\") call, so any message printed by\nparse_tree_gently() is superfluous.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex ccc529e5e..f9b51db7a 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,8 +143,6 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n-\tint gently = revs->ignore_missing_links ||\n-\t\t revs->exclude_promisor_objects;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -152,7 +150,7 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, gently) < 0) {\n+\tif (parse_tree_gently(tree, 1) < 0) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355656","messageId":"3c7272fb3d711a86572c9e9a620025e4b7a93394.1534291033.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534291033.git.matvore@google.com","subject":"[PATCH v5 4/6] rev-list: handle missing tree objects properly","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-15T00:22:53Z","receivedAt":"2018-08-15T00:23:15Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Previously, we assumed only blob objects could be missing. This patch\nmakes rev-list handle missing trees like missing blobs. The --missing=*\nand --exclude-promisor-objects flags now work for trees as they already\ndo for blobs. This is demonstrated in t6112.\n\nSigned-off-by: Matthew DeVore \n---\n builtin/rev-list.c | 11 ++++---\n list-objects.c | 11 +++++--\n revision.h | 15 +++++++++\n t/t0410-partial-clone.sh | 45 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 13 ++++++++\n t/t6112-rev-list-filters-objects.sh | 17 ++++++++++\n 6 files changed, 105 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 5b07f3f4a..49d6deed7 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -6,6 +6,7 @@\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"object.h\"\n #include \"object-store.h\"\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n@@ -209,7 +210,8 @@ static inline void finish_object__ma(struct object *obj)\n \t */\n \tswitch (arg_missing_action) {\n \tcase MA_ERROR:\n-\t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n+\t\tdie(\"missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tcase MA_ALLOW_ANY:\n@@ -222,8 +224,8 @@ static inline void finish_object__ma(struct object *obj)\n \tcase MA_ALLOW_PROMISOR:\n \t\tif (is_promisor_object(&obj->oid))\n \t\t\treturn;\n-\t\tdie(\"unexpected missing blob object '%s'\",\n-\t\t oid_to_hex(&obj->oid));\n+\t\tdie(\"unexpected missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tdefault:\n@@ -235,7 +237,7 @@ static inline void finish_object__ma(struct object *obj)\n static int finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\tif (!has_object_file(&obj->oid)) {\n \t\tfinish_object__ma(obj);\n \t\treturn 1;\n \t}\n@@ -373,6 +375,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tinit_revisions(&revs, prefix);\n \trevs.abbrev = DEFAULT_ABBREV;\n \trevs.commit_format = CMIT_FMT_UNSPECIFIED;\n+\trevs.do_not_die_on_missing_tree = 1;\n \n \t/*\n \t * Scan the argument list before invoking setup_revisions(), so that we\ndiff --git a/list-objects.c b/list-objects.c\nindex f9b51db7a..243192af5 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,6 +143,7 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tint failed_parse;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -150,7 +151,9 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, 1) < 0) {\n+\n+\tfailed_parse = parse_tree_gently(tree, 1);\n+\tif (failed_parse) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n@@ -163,7 +166,8 @@ static void process_tree(struct traversal_context *ctx,\n \t\t is_promisor_object(&obj->oid))\n \t\t\treturn;\n \n-\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n+\t\tif (!revs->do_not_die_on_missing_tree)\n+\t\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \n \tstrbuf_addstr(base, name);\n@@ -178,7 +182,8 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tprocess_tree_contents(ctx, tree, base);\n+\tif (!failed_parse)\n+\t\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\ndiff --git a/revision.h b/revision.h\nindex c599c34da..5118aaaa9 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -125,6 +125,21 @@ struct rev_info {\n \t\t\tline_level_traverse:1,\n \t\t\ttree_blobs_in_commit_order:1,\n \n+\t\t\t/*\n+\t\t\t * Blobs are shown without regard for their existence.\n+\t\t\t * But not so for trees: unless exclude_promisor_objects\n+\t\t\t * is set and the tree in question is a promisor object;\n+\t\t\t * OR ignore_missing_links is set, the revision walker\n+\t\t\t * dies with a \"bad tree object HASH\" message when\n+\t\t\t * encountering a missing tree. For callers that can\n+\t\t\t * handle missing trees and want them to be filterable\n+\t\t\t * and showable, set this to true. The revision walker\n+\t\t\t * will filter and show such a missing tree as usual,\n+\t\t\t * but will not attempt to recurse into this tree\n+\t\t\t * object.\n+\t\t\t */\n+\t\t\tdo_not_die_on_missing_tree:1,\n+\n \t\t\t/* for internal use only */\n \t\t\texclude_promisor_objects:1;\n \ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 4984ca583..a1b93c72c 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -186,6 +186,51 @@ test_expect_success 'rev-list stops traversal at missing and promised commit' '\n \t! grep $FOO out\n '\n \n+test_expect_success 'missing tree objects with --missing=allow-promisor and --exclude-promisor-objects' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo foo &&\n+\ttest_commit -C repo bar &&\n+\ttest_commit -C repo baz &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse bar^{tree}) &&\n+\tpromise_and_delete $(git -C repo rev-parse foo^{tree}) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-promisor --objects HEAD >objs 2>rev_list_err &&\n+\ttest_line_count = 0 rev_list_err &&\n+\t# 3 commits, 3 blobs, and 1 tree\n+\ttest_line_count = 7 objs &&\n+\n+\t# Do the same for --exclude-promisor-objects, but with all trees gone.\n+\tpromise_and_delete $(git -C repo rev-parse baz^{tree}) &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >objs 2>rev_list_err &&\n+\ttest_line_count = 0 rev_list_err &&\n+\t# 3 commits, no blobs or trees\n+\ttest_line_count = 3 objs\n+'\n+\n+test_expect_success 'missing non-root tree object and rev-list' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\tmkdir repo/dir &&\n+\techo foo > repo/dir/foo &&\n+\tgit -C repo add dir/foo &&\n+\tgit -C repo commit -m \"commit dir/foo\" &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse HEAD:dir) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n+\ttest_line_count = 0 rev_list_err &&\n+\t# 1 commit and 1 tree\n+\ttest_line_count = 2 objs\n+'\n+\n test_expect_success 'rev-list stops traversal at missing and promised tree' '\n \trm -rf repo &&\n \ttest_create_repo repo &&\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 6710c8bc8..5e35f33bf 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -59,6 +59,19 @@ test_expect_success 'verify normal and blob:none packfiles have same commits/tre\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'get an error for missing tree object' '\n+\tgit init r5 &&\n+\techo foo > r5/foo &&\n+\tgit -C r5 add foo &&\n+\tgit -C r5 commit -m \"foo\" &&\n+\tdel=$(git -C r5 rev-parse HEAD^{tree} | sed \"s|..|&/|\") &&\n+\trm r5/.git/objects/$del &&\n+\ttest_must_fail git -C r5 pack-objects --rev --stdout 2>bad_tree <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgrep -q \"bad tree object\" bad_tree\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 0a37dd5f9..fc0f92a16 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -196,6 +196,23 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for trees' '\n+\tTREE=$(git -C r3 rev-parse HEAD:dir1) &&\n+\n+\trm r3/.git/objects/$(echo $TREE | sed \"s|^..|&/|\") &&\n+\n+\tgit -C r3 rev-list --quiet --missing=print --objects HEAD >missing_objs 2>rev_list_err &&\n+\techo \"?$TREE\" >expected &&\n+\ttest_cmp expected missing_objs &&\n+\n+\t# do not complain when a missing tree cannot be parsed\n+\ttest_line_count = 0 rev_list_err &&\n+\n+\tgit -C r3 rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n+\t! grep $TREE objs &&\n+\ttest_line_count = 0 rev_list_err\n+'\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355657","messageId":"88556df54c5f7cdd6c0005923f0c27ab3bf3280e.1534291033.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534291033.git.matvore@google.com","subject":"[PATCH v5 5/6] revision: mark non-user-given objects instead","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-15T00:22:54Z","receivedAt":"2018-08-15T00:23:17Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Currently, list-objects.c incorrectly treats all root trees of commits\nas USER_GIVEN. Also, it would be easier to mark objects that are\nnon-user-given instead of user-given, since the places in the code\nwhere we access an object through a reference are more obvious than\nthe places where we access an object that was given by the user.\n\nResolve these two problems by introducing a flag NOT_USER_GIVEN that\nmarks blobs and trees that are non-user-given, replacing USER_GIVEN.\n(Only blobs and trees are marked because this mark is only used when\nfiltering objects, and filtering of other types of objects is not\nsupported yet.)\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 31 ++++++++++++++++++-------------\n revision.c | 1 -\n revision.h | 10 +++++++---\n 3 files changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 243192af5..7a1a0929d 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -53,7 +53,7 @@ static void process_blob(struct traversal_context *ctx,\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n \t\t\t\t path->buf, &path->buf[pathlen],\n \t\t\t\t ctx->filter_data);\n@@ -120,17 +120,19 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\tcontinue;\n \t\t}\n \n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\tstruct tree *t = lookup_tree(the_repository, entry.oid);\n+\t\t\tt->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_tree(ctx, t, base, entry.path);\n+\t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid->hash,\n \t\t\t\t\tbase, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\telse {\n+\t\t\tstruct blob *b = lookup_blob(the_repository, entry.oid);\n+\t\t\tb->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_blob(ctx, b, base, entry.path);\n+\t\t}\n \t}\n }\n \n@@ -171,7 +173,7 @@ static void process_tree(struct traversal_context *ctx,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -185,7 +187,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (!failed_parse)\n \t\tprocess_tree_contents(ctx, tree, base);\n \n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -301,8 +303,11 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n-\t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tif (get_commit_tree(commit)) {\n+\t\t\tstruct tree *tree = get_commit_tree(commit);\n+\t\t\ttree->object.flags |= NOT_USER_GIVEN;\n+\t\t\tadd_pending_tree(ctx->revs, tree);\n+\t\t}\n \t\tctx->show_commit(commit, ctx->show_data);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\ndiff --git a/revision.c b/revision.c\nindex 062749437..6d355b43c 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -175,7 +175,6 @@ static void add_pending_object_with_path(struct rev_info *revs,\n \t\tstrbuf_release(&buf);\n \t\treturn; /* do not add the commit itself */\n \t}\n-\tobj->flags |= USER_GIVEN;\n \tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n }\n \ndiff --git a/revision.h b/revision.h\nindex 5118aaaa9..2d381e636 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -8,7 +8,11 @@\n #include \"diff.h\"\n #include \"commit-slab-decl.h\"\n \n-/* Remember to update object flag allocation in object.h */\n+/* Remember to update object flag allocation in object.h\n+ * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n+ * filtering trees and blobs, but it may be useful to support filtering commits\n+ * in the future.\n+ */\n #define SEEN\t\t(1u<<0)\n #define UNINTERESTING (1u<<1)\n #define TREESAME\t(1u<<2)\n@@ -20,9 +24,9 @@\n #define SYMMETRIC_LEFT\t(1u<<8)\n #define PATCHSAME\t(1u<<9)\n #define BOTTOM\t\t(1u<<10)\n-#define USER_GIVEN\t(1u<<25) /* given directly by the user */\n+#define NOT_USER_GIVEN\t(1u<<25) /* tree or blob not given directly by user */\n #define TRACK_LINEAR\t(1u<<26)\n-#define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n+#define ALL_REV_FLAGS\t(((1u<<11)-1) | NOT_USER_GIVEN | TRACK_LINEAR)\n \n #define DECORATE_SHORT_REFS\t1\n #define DECORATE_FULL_REFS\t2\n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355658","messageId":"70c09e892e0acf71ecb0c93ff3cad09f5be5f19d.1534291033.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534291033.git.matvore@google.com","subject":"[PATCH v5 6/6] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-15T00:22:55Z","receivedAt":"2018-08-15T00:23:20Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Teach list-objects the \"tree:0\" filter which allows for filtering\nout all tree and blob objects (unless other objects are explicitly\nspecified by the user). The purpose of this patch is to allow smaller\npartial clones.\n\nThe name of this filter - tree:0 - does not explicitly specify that\nit also filters out all blobs, but this should not cause much confusion\nbecause blobs are not at all useful without the trees that refer to\nthem.\n\nI also consider only:commits as a name, but this is inaccurate because\nit suggests that annotated tags are omitted, but actually they are\nincluded.\n\nThe name \"tree:0\" allows later filtering based on depth, i.e. \"tree:1\"\nwould filter out all but the root tree and blobs. In order to avoid\nconfusion between 0 and capital O, the documentation was worded in a\nsomewhat round-about way that also hints at this future improvement to\nthe feature.\n\nSigned-off-by: Matthew DeVore \n---\n Documentation/rev-list-options.txt | 5 +++\n list-objects-filter-options.c | 4 +++\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 50 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 28 +++++++++++++++\n t/t5616-partial-clone.sh | 38 ++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 12 +++++++\n 7 files changed, 138 insertions(+)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 7b273635d..5f1672913 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -731,6 +731,11 @@ the requested refs.\n +\n The form '--filter=sparse:path=' similarly uses a sparse-checkout\n specification contained in .\n++\n+The form '--filter=tree:' omits all blobs and trees whose depth\n+from the root tree is >= (minimum depth if an object is located\n+at multiple depths in the commits traversed). Currently, only =0\n+is supported, which omits all blobs and trees.\n \n --no-filter::\n \tTurn off any previous `--filter=` argument.\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex c0e2bd6a0..a28382940 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -50,6 +50,10 @@ static int gently_parse_list_objects_filter(\n \t\t\treturn 0;\n \t\t}\n \n+\t} else if (!strcmp(arg, \"tree:0\")) {\n+\t\tfilter_options->choice = LOFC_TREE_NONE;\n+\t\treturn 0;\n+\n \t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n \t\tstruct object_context oc;\n \t\tstruct object_id sparse_oid;\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 0000a61f8..af64e5c66 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -10,6 +10,7 @@ enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n+\tLOFC_TREE_NONE,\n \tLOFC_SPARSE_OID,\n \tLOFC_SPARSE_PATH,\n \tLOFC__COUNT /* must be last */\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex a0ba78b20..8e3caf5bf 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -80,6 +80,55 @@ static void *filter_blobs_none__init(\n \treturn d;\n }\n \n+/*\n+ * A filter for list-objects to omit ALL trees and blobs from the traversal.\n+ * Can OPTIONALLY collect a list of the omitted OIDs.\n+ */\n+struct filter_trees_none_data {\n+\tstruct oidset *omits;\n+};\n+\n+static enum list_objects_filter_result filter_trees_none(\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_trees_none_data *filter_data = filter_data_;\n+\n+\tswitch (filter_situation) {\n+\tdefault:\n+\t\tdie(\"unknown filter_situation\");\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFS_BEGIN_TREE:\n+\tcase LOFS_BLOB:\n+\t\tif (filter_data->omits)\n+\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n+\n+\tcase LOFS_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\treturn LOFR_ZERO;\n+\n+\t}\n+}\n+\n+static void* filter_trees_none__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn)\n+{\n+\tstruct filter_trees_none_data *d = xcalloc(1, sizeof(*d));\n+\td->omits = omitted;\n+\n+\t*filter_fn = filter_trees_none;\n+\t*filter_free_fn = free;\n+\treturn d;\n+}\n+\n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n@@ -374,6 +423,7 @@ static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n+\tfilter_trees_none__init,\n \tfilter_sparse_oid__init,\n \tfilter_sparse_path__init,\n };\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 5e35f33bf..985668a07 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -72,6 +72,34 @@ test_expect_success 'get an error for missing tree object' '\n \tgrep -q \"bad tree object\" bad_tree\n '\n \n+test_expect_success 'setup for tests of tree:0' '\n+\tmkdir r1/subtree &&\n+\techo \"This is a file in a subtree\" >r1/subtree/file &&\n+\tgit -C r1 add subtree/file &&\n+\tgit -C r1 commit -m subtree\n+'\n+\n+test_expect_success 'verify tree:0 packfile has no blobs or trees' '\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\t! grep -E \"tree|blob\" objs\n+'\n+\n+test_expect_success 'grab tree directly when using tree:0' '\n+\t# We should get the tree specified directly but not its blobs or subtrees.\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD:\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\tawk -e \"/tree|blob/{print \\$1}\" objs >trees_and_blobs &&\n+\tgit -C r1 rev-parse HEAD: >expected &&\n+\ttest_cmp trees_and_blobs expected\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex bbbe7537d..8eeb85fbc 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -154,6 +154,44 @@ test_expect_success 'partial clone with transfer.fsckobjects=1 uses index-pack -\n \tgrep \"git index-pack.*--fsck-objects\" trace\n '\n \n+test_expect_success 'use fsck before and after manually fetching a missing subtree' '\n+\t# push new commit so server has a subtree\n+\tmkdir src/dir &&\n+\techo \"in dir\" >src/dir/file.txt &&\n+\tgit -C src add dir/file.txt &&\n+\tgit -C src commit -m \"file in dir\" &&\n+\tgit -C src push -u srv master &&\n+\tSUBTREE=$(git -C src rev-parse HEAD:dir) &&\n+\n+\trm -rf dst &&\n+\tgit clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n+\tgit -C dst fsck &&\n+\n+\t# Make sure we only have commits, and all trees and blobs are missing.\n+\tgit -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n+\tawk -f print_1.awk fetched_objects \\\n+\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n+\tsort fetched_types -u >unique_types.observed &&\n+\techo commit >unique_types.expected &&\n+\ttest_cmp unique_types.observed unique_types.expected &&\n+\n+\t# Auto-fetch a tree with cat-file.\n+\tgit -C dst cat-file -p $SUBTREE >tree_contents &&\n+\tgrep file.txt tree_contents &&\n+\n+\t# fsck still works after an auto-fetch of a tree.\n+\tgit -C dst fsck &&\n+\n+\t# Auto-fetch all remaining trees and blobs with --missing=error\n+\tgit -C dst rev-list master --missing=error --objects >fetched_objects &&\n+\ttest_line_count = 70 fetched_objects &&\n+\tawk -f print_1.awk fetched_objects \\\n+\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n+\tsort fetched_types -u >unique_types.observed &&\n+\tprintf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n+\ttest_cmp unique_types.observed unique_types.expected\n+'\n+\n test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' '\n \trm -rf src dst &&\n \tgit init src &&\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex fc0f92a16..30bf1c73e 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -213,6 +213,18 @@ test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for tre\n \ttest_line_count = 0 rev_list_err\n '\n \n+# Test tree:0 filter.\n+\n+test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n+\tgit -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=tree:0 \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed s/~// \\\n+\t\t| xargs -n1 git -C r3 cat-file -t \\\n+\t\t| sort -u >filtered_types &&\n+\tprintf \"blob\\ntree\\n\" > expected &&\n+\ttest_cmp filtered_types expected\n+'\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355660","messageId":"20180815012224.GA4745@sigill.intra.peff.net","threadId":"49086","inReplyTo":"CAMfpvhJM2hUXTfTgY9jA4aB+bgx44xbCktCacpy_yWacyjk00w@mail.gmail.com","subject":"Re: [PATCH v4 6/6] list-objects-filter: implement filter tree:0","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-08-15T01:22:24Z","receivedAt":"2018-08-15T01:22:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 14, 2018 at 04:55:34PM -0700, Matthew DeVore wrote:\n\n> > - it's expensive to compute, because you have to actually walk all of\n> > the possible commits and trees that could refer to it. This\n> > prohibits a lot of other optimizations like reachability bitmaps\n> > (though with some complexity you could cache the depths, too).\n> I think what the user likely wants is to use the minimum depth based\n> on the commits in the traversal, not every commit in the repo - is\n> this what you mean?\n\nRight, I'd agree they probably want the minimum for that traversal. And\nfor `rev-list --filter`, that's probably OK. But keep in mind the main\ngoal for --filter is using it for fetches, and many servers do not\nperform the traversal at all. Instead they use reachability bitmaps to\ncome up with the set of objects to send. The bitmaps have enough\ninformation to say \"remove all trees from the set\", but not enough to do\nany kind of depth-based calculation (not even \"is this a root tree\").\n\n> Makes sense. I changed it like this -\n> \n> diff --git a/Documentation/rev-list-options.txt\n> b/Documentation/rev-list-options.txt\n> index 0b5f77ad3..5f1672913 100644\n> --- a/Documentation/rev-list-options.txt\n> +++ b/Documentation/rev-list-options.txt\n> @@ -732,8 +732,10 @@ the requested refs.\n> The form '--filter=sparse:path=' similarly uses a sparse-checkout\n> specification contained in .\n> +\n> -The form '--filter=tree:' omits all blobs and trees deeper than\n> - from the root tree. Currently, only =0 is supported.\n> +The form '--filter=tree:' omits all blobs and trees whose depth\n> +from the root tree is >= (minimum depth if an object is located\n> +at multiple depths in the commits traversed). Currently, only =0\n> +is supported, which omits all blobs and trees.\n\nYes, I think that makes sense. Thanks!\n\n-Peff\n"},{"id":"355695","messageId":"xmqqva8bk3o4.fsf@gitster-ct.c.googlers.com","threadId":"49086","inReplyTo":"CAMfpvh+H5UeHmCqeVNZNoOe_-vgH3_3aTuO+c=imgzaVkC6N2g@mail.gmail.com","subject":"Re: [PATCH v4 6/6] list-objects-filter: implement filter tree:0","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-08-15T16:14:35Z","receivedAt":"2018-08-15T16:14:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthew DeVore writes:\n\n> Thank you. I changed it to this:\n> awk -e \"/tree|blob/{print \\$1}\" objs >trees_and_blobs\n\nThe \"-e\" option does not appear in\n\nhttp://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html\n\nand I think you can safely drop it from your command line.\n\n If no -f option is specified, the first operand to awk shall be the\n text of the awk program. The application shall supply the program\n operand as a single argument to awk. If the text does not end in a\n , awk shall interpret the text as if it did.\n\n"},{"id":"355696","messageId":"xmqqr2izk3j0.fsf@gitster-ct.c.googlers.com","threadId":"49086","inReplyTo":"20180815012224.GA4745@sigill.intra.peff.net","subject":"Re: [PATCH v4 6/6] list-objects-filter: implement filter tree:0","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-08-15T16:17:39Z","receivedAt":"2018-08-15T16:17:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> Right, I'd agree they probably want the minimum for that traversal. And\n> for `rev-list --filter`, that's probably OK. But keep in mind the main\n> goal for --filter is using it for fetches, and many servers do not\n> perform the traversal at all. Instead they use reachability bitmaps to\n> come up with the set of objects to send. The bitmaps have enough\n> information to say \"remove all trees from the set\", but not enough to do\n> any kind of depth-based calculation (not even \"is this a root tree\").\n\nIf the depth-based cutoff turns out to make sense (on which I\nhaven't formed an opinion yet), newer version of pack bitmaps could\nstore that information ;-)\n\nHow are these \"fitler\" expressions negotiated between the fetcher\nand uploader? Does a \"fetch-patch\" say \"am I allowed to ask you to\nfilter with tree:4?\" and refrain from using the option when\n\"upload-pack\" says \"no\"?\n"},{"id":"355703","messageId":"CAMfpvh+F0asn94+M0369MoCaBKVoVFMzKarVm2kk0jaD-AdaGw@mail.gmail.com","threadId":"49086","inReplyTo":"xmqqva8bk3o4.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 6/6] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-15T16:37:24Z","receivedAt":"2018-08-15T16:37:38Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Wed, Aug 15, 2018 at 9:14 AM Junio C Hamano wrote:\n>\n> Matthew DeVore writes:\n>\n> > Thank you. I changed it to this:\n> > awk -e \"/tree|blob/{print \\$1}\" objs >trees_and_blobs\n>\n> The \"-e\" option does not appear in\n>\n> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html\n>\n> and I think you can safely drop it from your command line.\nFixed it, thank you. It will be in the next patchset version.\n\n>\n> If no -f option is specified, the first operand to awk shall be the\n> text of the awk program. The application shall supply the program\n> operand as a single argument to awk. If the text does not end in a\n> , awk shall interpret the text as if it did.\n>\n"},{"id":"355729","messageId":"CAMfpvhJA3h8cFB_Weh_cDdqQBHccYfCrjs-S-T_nyDG=JVi6Tg@mail.gmail.com","threadId":"49086","inReplyTo":"xmqqr2izk3j0.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 6/6] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-15T17:54:24Z","receivedAt":"2018-08-15T17:54:38Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Wed, Aug 15, 2018 at 9:17 AM Junio C Hamano wrote:\n>\n> Jeff King writes:\n>\n> > Right, I'd agree they probably want the minimum for that traversal. And\n> > for `rev-list --filter`, that's probably OK. But keep in mind the main\n> > goal for --filter is using it for fetches, and many servers do not\n> > perform the traversal at all. Instead they use reachability bitmaps to\n> > come up with the set of objects to send. The bitmaps have enough\n> > information to say \"remove all trees from the set\", but not enough to do\n> > any kind of depth-based calculation (not even \"is this a root tree\").\n>\n> If the depth-based cutoff turns out to make sense (on which I\n> haven't formed an opinion yet), newer version of pack bitmaps could\n> store that information ;-)\n>\n> How are these \"fitler\" expressions negotiated between the fetcher\n> and uploader? Does a \"fetch-patch\" say \"am I allowed to ask you to\n> filter with tree:4?\" and refrain from using the option when\n> \"upload-pack\" says \"no\"?\n\nI couldn't find a feature like that for the existing features, but\nadding such a think seems reasonable to me. (thinking in terms of\nprotocol v2,) There could be a filter-check command which takes a\nsingle argument: the filter string (like \"tree:4\"), and responds with\na single line: either \"ok\" or \"unsupported\".\n"},{"id":"355779","messageId":"cover.1534374650.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"[PATCH v6 0/6] filter: support for excluding all trees and blobs","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-15T23:19:05Z","receivedAt":"2018-08-15T23:23:09Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Applied suggestion from Junio about removing -e flag from awk invocation.\nSending an updated patchset now since I haven't heard any other comments for a\nwhile, and I don't believe Jonathan, the most active reviewer, has any more\nconcerns.\n\nMatthew DeVore (6):\n list-objects: store common func args in struct\n list-objects: refactor to process_tree_contents\n list-objects: always parse trees gently\n rev-list: handle missing tree objects properly\n revision: mark non-user-given objects instead\n list-objects-filter: implement filter tree:0\n\n Documentation/rev-list-options.txt | 5 +\n builtin/rev-list.c | 11 +-\n list-objects-filter-options.c | 4 +\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 50 ++++++\n list-objects.c | 232 +++++++++++++------------\n revision.c | 1 -\n revision.h | 25 ++-\n t/t0410-partial-clone.sh | 45 +++++\n t/t5317-pack-objects-filter-objects.sh | 41 +++++\n t/t5616-partial-clone.sh | 38 ++++\n t/t6112-rev-list-filters-objects.sh | 29 ++++\n 12 files changed, 364 insertions(+), 118 deletions(-)\n\n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355780","messageId":"aaf4c2d93e25aae32275f8d58b04a44b599c6323.1534374650.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534374650.git.matvore@google.com","subject":"[PATCH v6 1/6] list-objects: store common func args in struct","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-15T23:19:06Z","receivedAt":"2018-08-15T23:23:19Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will make utility functions easier to create, as done by the next\npatch.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 158 +++++++++++++++++++++++--------------------------\n 1 file changed, 74 insertions(+), 84 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex c99c47ac1..584518a3f 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -12,20 +12,25 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n \n-static void process_blob(struct rev_info *revs,\n+struct traversal_context {\n+\tstruct rev_info *revs;\n+\tshow_object_fn show_object;\n+\tshow_commit_fn show_commit;\n+\tvoid *show_data;\n+\tfilter_object_fn filter_fn;\n+\tvoid *filter_data;\n+};\n+\n+static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \n-\tif (!revs->blob_objects)\n+\tif (!ctx->revs->blob_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad blob object\");\n@@ -41,21 +46,21 @@ static void process_blob(struct rev_info *revs,\n \t * may cause the actual filter to report an incomplete list\n \t * of missing objects.\n \t */\n-\tif (revs->exclude_promisor_objects &&\n+\tif (ctx->revs->exclude_promisor_objects &&\n \t !has_object_file(&obj->oid) &&\n \t is_promisor_object(&obj->oid))\n \t\treturn;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BLOB, obj,\n-\t\t\t path->buf, &path->buf[pathlen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n+\t\t\t\t path->buf, &path->buf[pathlen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, path->buf, cb_data);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -81,26 +86,21 @@ static void process_blob(struct rev_info *revs,\n * the link, and how to do it. Whether it necessarily makes\n * any sense what-so-ever to ever do that is another issue.\n */\n-static void process_gitlink(struct rev_info *revs,\n+static void process_gitlink(struct traversal_context *ctx,\n \t\t\t const unsigned char *sha1,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data)\n+\t\t\t const char *name)\n {\n \t/* Nothing to do */\n }\n \n-static void process_tree(struct rev_info *revs,\n+static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &tree->object;\n+\tstruct rev_info *revs = ctx->revs;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n@@ -133,14 +133,14 @@ static void process_tree(struct rev_info *revs,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BEGIN_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, base->buf, cb_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -157,29 +157,25 @@ static void process_tree(struct rev_info *revs,\n \t\t}\n \n \t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(revs,\n+\t\t\tprocess_tree(ctx,\n \t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(revs, entry.oid->hash,\n-\t\t\t\t\tshow, base, entry.path,\n-\t\t\t\t\tcb_data);\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n \t\telse\n-\t\t\tprocess_blob(revs,\n+\t\t\tprocess_blob(ctx,\n \t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t}\n \n-\tif (!(obj->flags & USER_GIVEN) && filter_fn) {\n-\t\tr = filter_fn(LOFS_END_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \t\tif (r & LOFR_MARK_SEEN)\n \t\t\tobj->flags |= SEEN;\n \t\tif (r & LOFR_DO_SHOW)\n-\t\t\tshow(obj, base->buf, cb_data);\n+\t\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \t}\n \n \tstrbuf_setlen(base, baselen);\n@@ -242,19 +238,15 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n-static void traverse_trees_and_blobs(struct rev_info *revs,\n-\t\t\t\t struct strbuf *base,\n-\t\t\t\t show_object_fn show_object,\n-\t\t\t\t void *show_data,\n-\t\t\t\t filter_object_fn filter_fn,\n-\t\t\t\t void *filter_data)\n+static void traverse_trees_and_blobs(struct traversal_context *ctx,\n+\t\t\t\t struct strbuf *base)\n {\n \tint i;\n \n \tassert(base->len == 0);\n \n-\tfor (i = 0; i < revs->pending.nr; i++) {\n-\t\tstruct object_array_entry *pending = revs->pending.objects + i;\n+\tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n+\t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n@@ -262,62 +254,49 @@ static void traverse_trees_and_blobs(struct rev_info *revs,\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tshow_object(obj, name, show_data);\n+\t\t\tctx->show_object(obj, name, ctx->show_data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(revs, (struct tree *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(revs, (struct blob *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n \t\t oid_to_hex(&obj->oid), name);\n \t}\n-\tobject_array_clear(&revs->pending);\n+\tobject_array_clear(&ctx->revs->pending);\n }\n \n-static void do_traverse(struct rev_info *revs,\n-\t\t\tshow_commit_fn show_commit,\n-\t\t\tshow_object_fn show_object,\n-\t\t\tvoid *show_data,\n-\t\t\tfilter_object_fn filter_fn,\n-\t\t\tvoid *filter_data)\n+static void do_traverse(struct traversal_context *ctx)\n {\n \tstruct commit *commit;\n \tstruct strbuf csp; /* callee's scratch pad */\n \tstrbuf_init(&csp, PATH_MAX);\n \n-\twhile ((commit = get_revision(revs)) != NULL) {\n+\twhile ((commit = get_revision(ctx->revs)) != NULL) {\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n \t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(revs, get_commit_tree(commit));\n-\t\tshow_commit(commit, show_data);\n+\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tctx->show_commit(commit, ctx->show_data);\n \n-\t\tif (revs->tree_blobs_in_commit_order)\n+\t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n \t\t\t * NEEDSWORK: Adding the tree and then flushing it here\n \t\t\t * needs a reallocation for each commit. Can we pass the\n \t\t\t * tree directory without allocation churn?\n \t\t\t */\n-\t\t\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t\t\t show_object, show_data,\n-\t\t\t\t\t\t filter_fn, filter_data);\n+\t\t\ttraverse_trees_and_blobs(ctx, &csp);\n \t}\n-\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t show_object, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\ttraverse_trees_and_blobs(ctx, &csp);\n \tstrbuf_release(&csp);\n }\n \n@@ -326,7 +305,14 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t\t show_object_fn show_object,\n \t\t\t void *show_data)\n {\n-\tdo_traverse(revs, show_commit, show_object, show_data, NULL, NULL);\n+\tstruct traversal_context ctx;\n+\tctx.revs = revs;\n+\tctx.show_commit = show_commit;\n+\tctx.show_object = show_object;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\tctx.filter_data = NULL;\n+\tdo_traverse(&ctx);\n }\n \n void traverse_commit_list_filtered(\n@@ -337,14 +323,18 @@ void traverse_commit_list_filtered(\n \tvoid *show_data,\n \tstruct oidset *omitted)\n {\n-\tfilter_object_fn filter_fn = NULL;\n+\tstruct traversal_context ctx;\n \tfilter_free_fn filter_free_fn = NULL;\n-\tvoid *filter_data = NULL;\n-\n-\tfilter_data = list_objects_filter__init(omitted, filter_options,\n-\t\t\t\t\t\t&filter_fn, &filter_free_fn);\n-\tdo_traverse(revs, show_commit, show_object, show_data,\n-\t\t filter_fn, filter_data);\n-\tif (filter_data && filter_free_fn)\n-\t\tfilter_free_fn(filter_data);\n+\n+\tctx.revs = revs;\n+\tctx.show_object = show_object;\n+\tctx.show_commit = show_commit;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\n+\tctx.filter_data = list_objects_filter__init(omitted, filter_options,\n+\t\t\t\t\t\t &ctx.filter_fn, &filter_free_fn);\n+\tdo_traverse(&ctx);\n+\tif (ctx.filter_data && filter_free_fn)\n+\t\tfilter_free_fn(ctx.filter_data);\n }\n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355781","messageId":"55da169659b3c04e3f38f876744ee76eb6d0db93.1534374650.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534374650.git.matvore@google.com","subject":"[PATCH v6 3/6] list-objects: always parse trees gently","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-15T23:19:08Z","receivedAt":"2018-08-15T23:23:19Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"If parsing fails when revs->ignore_missing_links and\nrevs->exclude_promisor_objects are both false, we print the OID anyway\nin the die(\"bad tree object...\") call, so any message printed by\nparse_tree_gently() is superfluous.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex ccc529e5e..f9b51db7a 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,8 +143,6 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n-\tint gently = revs->ignore_missing_links ||\n-\t\t revs->exclude_promisor_objects;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -152,7 +150,7 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, gently) < 0) {\n+\tif (parse_tree_gently(tree, 1) < 0) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355782","messageId":"f753322f09757bdafe62dae89fa53171956ed69e.1534374650.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534374650.git.matvore@google.com","subject":"[PATCH v6 2/6] list-objects: refactor to process_tree_contents","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-15T23:19:07Z","receivedAt":"2018-08-15T23:23:19Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will be used in a follow-up patch to reduce indentation needed when\ninvoking the logic conditionally. i.e. rather than:\n\nif (foo) {\n\twhile (...) {\n\t\t/* this is very indented */\n\t}\n}\n\nwe will have:\n\nif (foo)\n\tprocess_tree_contents(...);\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 68 ++++++++++++++++++++++++++++++--------------------\n 1 file changed, 41 insertions(+), 27 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 584518a3f..ccc529e5e 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -94,6 +94,46 @@ static void process_gitlink(struct traversal_context *ctx,\n \t/* Nothing to do */\n }\n \n+static void process_tree(struct traversal_context *ctx,\n+\t\t\t struct tree *tree,\n+\t\t\t struct strbuf *base,\n+\t\t\t const char *name);\n+\n+static void process_tree_contents(struct traversal_context *ctx,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *base)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tenum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ?\n+\t\tall_entries_interesting : entry_not_interesting;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tif (match != all_entries_interesting) {\n+\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n+\t\t\t\t\t\t &ctx->revs->diffopt.pathspec);\n+\t\t\tif (match == all_entries_not_interesting)\n+\t\t\t\tbreak;\n+\t\t\tif (match == entry_not_interesting)\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (S_ISDIR(entry.mode))\n+\t\t\tprocess_tree(ctx,\n+\t\t\t\t lookup_tree(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t\telse if (S_ISGITLINK(entry.mode))\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash,\n+\t\t\t\t\tbase, entry.path);\n+\t\telse\n+\t\t\tprocess_blob(ctx,\n+\t\t\t\t lookup_blob(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t}\n+}\n+\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n@@ -101,10 +141,6 @@ static void process_tree(struct traversal_context *ctx,\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n-\tstruct tree_desc desc;\n-\tstruct name_entry entry;\n-\tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n-\t\tall_entries_interesting: entry_not_interesting;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \tint gently = revs->ignore_missing_links ||\n@@ -144,29 +180,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tinit_tree_desc(&desc, tree->buffer, tree->size);\n-\n-\twhile (tree_entry(&desc, &entry)) {\n-\t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n-\t\t\t\t\t\t &revs->diffopt.pathspec);\n-\t\t\tif (match == all_entries_not_interesting)\n-\t\t\t\tbreak;\n-\t\t\tif (match == entry_not_interesting)\n-\t\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t}\n+\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355783","messageId":"22b1e39b90de3adff3b5ae6ba3afb0aac7c4acbb.1534374650.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534374650.git.matvore@google.com","subject":"[PATCH v6 4/6] rev-list: handle missing tree objects properly","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-15T23:19:09Z","receivedAt":"2018-08-15T23:23:20Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Previously, we assumed only blob objects could be missing. This patch\nmakes rev-list handle missing trees like missing blobs. The --missing=*\nand --exclude-promisor-objects flags now work for trees as they already\ndo for blobs. This is demonstrated in t6112.\n\nSigned-off-by: Matthew DeVore \n---\n builtin/rev-list.c | 11 ++++---\n list-objects.c | 11 +++++--\n revision.h | 15 +++++++++\n t/t0410-partial-clone.sh | 45 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 13 ++++++++\n t/t6112-rev-list-filters-objects.sh | 17 ++++++++++\n 6 files changed, 105 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 5b07f3f4a..49d6deed7 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -6,6 +6,7 @@\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"object.h\"\n #include \"object-store.h\"\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n@@ -209,7 +210,8 @@ static inline void finish_object__ma(struct object *obj)\n \t */\n \tswitch (arg_missing_action) {\n \tcase MA_ERROR:\n-\t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n+\t\tdie(\"missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tcase MA_ALLOW_ANY:\n@@ -222,8 +224,8 @@ static inline void finish_object__ma(struct object *obj)\n \tcase MA_ALLOW_PROMISOR:\n \t\tif (is_promisor_object(&obj->oid))\n \t\t\treturn;\n-\t\tdie(\"unexpected missing blob object '%s'\",\n-\t\t oid_to_hex(&obj->oid));\n+\t\tdie(\"unexpected missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tdefault:\n@@ -235,7 +237,7 @@ static inline void finish_object__ma(struct object *obj)\n static int finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\tif (!has_object_file(&obj->oid)) {\n \t\tfinish_object__ma(obj);\n \t\treturn 1;\n \t}\n@@ -373,6 +375,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tinit_revisions(&revs, prefix);\n \trevs.abbrev = DEFAULT_ABBREV;\n \trevs.commit_format = CMIT_FMT_UNSPECIFIED;\n+\trevs.do_not_die_on_missing_tree = 1;\n \n \t/*\n \t * Scan the argument list before invoking setup_revisions(), so that we\ndiff --git a/list-objects.c b/list-objects.c\nindex f9b51db7a..243192af5 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,6 +143,7 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tint failed_parse;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -150,7 +151,9 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, 1) < 0) {\n+\n+\tfailed_parse = parse_tree_gently(tree, 1);\n+\tif (failed_parse) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n@@ -163,7 +166,8 @@ static void process_tree(struct traversal_context *ctx,\n \t\t is_promisor_object(&obj->oid))\n \t\t\treturn;\n \n-\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n+\t\tif (!revs->do_not_die_on_missing_tree)\n+\t\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \n \tstrbuf_addstr(base, name);\n@@ -178,7 +182,8 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tprocess_tree_contents(ctx, tree, base);\n+\tif (!failed_parse)\n+\t\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\ndiff --git a/revision.h b/revision.h\nindex c599c34da..5118aaaa9 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -125,6 +125,21 @@ struct rev_info {\n \t\t\tline_level_traverse:1,\n \t\t\ttree_blobs_in_commit_order:1,\n \n+\t\t\t/*\n+\t\t\t * Blobs are shown without regard for their existence.\n+\t\t\t * But not so for trees: unless exclude_promisor_objects\n+\t\t\t * is set and the tree in question is a promisor object;\n+\t\t\t * OR ignore_missing_links is set, the revision walker\n+\t\t\t * dies with a \"bad tree object HASH\" message when\n+\t\t\t * encountering a missing tree. For callers that can\n+\t\t\t * handle missing trees and want them to be filterable\n+\t\t\t * and showable, set this to true. The revision walker\n+\t\t\t * will filter and show such a missing tree as usual,\n+\t\t\t * but will not attempt to recurse into this tree\n+\t\t\t * object.\n+\t\t\t */\n+\t\t\tdo_not_die_on_missing_tree:1,\n+\n \t\t\t/* for internal use only */\n \t\t\texclude_promisor_objects:1;\n \ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 4984ca583..a1b93c72c 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -186,6 +186,51 @@ test_expect_success 'rev-list stops traversal at missing and promised commit' '\n \t! grep $FOO out\n '\n \n+test_expect_success 'missing tree objects with --missing=allow-promisor and --exclude-promisor-objects' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo foo &&\n+\ttest_commit -C repo bar &&\n+\ttest_commit -C repo baz &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse bar^{tree}) &&\n+\tpromise_and_delete $(git -C repo rev-parse foo^{tree}) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-promisor --objects HEAD >objs 2>rev_list_err &&\n+\ttest_line_count = 0 rev_list_err &&\n+\t# 3 commits, 3 blobs, and 1 tree\n+\ttest_line_count = 7 objs &&\n+\n+\t# Do the same for --exclude-promisor-objects, but with all trees gone.\n+\tpromise_and_delete $(git -C repo rev-parse baz^{tree}) &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >objs 2>rev_list_err &&\n+\ttest_line_count = 0 rev_list_err &&\n+\t# 3 commits, no blobs or trees\n+\ttest_line_count = 3 objs\n+'\n+\n+test_expect_success 'missing non-root tree object and rev-list' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\tmkdir repo/dir &&\n+\techo foo > repo/dir/foo &&\n+\tgit -C repo add dir/foo &&\n+\tgit -C repo commit -m \"commit dir/foo\" &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse HEAD:dir) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n+\ttest_line_count = 0 rev_list_err &&\n+\t# 1 commit and 1 tree\n+\ttest_line_count = 2 objs\n+'\n+\n test_expect_success 'rev-list stops traversal at missing and promised tree' '\n \trm -rf repo &&\n \ttest_create_repo repo &&\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 6710c8bc8..5e35f33bf 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -59,6 +59,19 @@ test_expect_success 'verify normal and blob:none packfiles have same commits/tre\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'get an error for missing tree object' '\n+\tgit init r5 &&\n+\techo foo > r5/foo &&\n+\tgit -C r5 add foo &&\n+\tgit -C r5 commit -m \"foo\" &&\n+\tdel=$(git -C r5 rev-parse HEAD^{tree} | sed \"s|..|&/|\") &&\n+\trm r5/.git/objects/$del &&\n+\ttest_must_fail git -C r5 pack-objects --rev --stdout 2>bad_tree <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgrep -q \"bad tree object\" bad_tree\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 0a37dd5f9..fc0f92a16 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -196,6 +196,23 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for trees' '\n+\tTREE=$(git -C r3 rev-parse HEAD:dir1) &&\n+\n+\trm r3/.git/objects/$(echo $TREE | sed \"s|^..|&/|\") &&\n+\n+\tgit -C r3 rev-list --quiet --missing=print --objects HEAD >missing_objs 2>rev_list_err &&\n+\techo \"?$TREE\" >expected &&\n+\ttest_cmp expected missing_objs &&\n+\n+\t# do not complain when a missing tree cannot be parsed\n+\ttest_line_count = 0 rev_list_err &&\n+\n+\tgit -C r3 rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n+\t! grep $TREE objs &&\n+\ttest_line_count = 0 rev_list_err\n+'\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355784","messageId":"df2c0f757b02e65775fa629f501dfde49ff2e050.1534374650.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534374650.git.matvore@google.com","subject":"[PATCH v6 5/6] revision: mark non-user-given objects instead","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-15T23:19:10Z","receivedAt":"2018-08-15T23:23:21Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Currently, list-objects.c incorrectly treats all root trees of commits\nas USER_GIVEN. Also, it would be easier to mark objects that are\nnon-user-given instead of user-given, since the places in the code\nwhere we access an object through a reference are more obvious than\nthe places where we access an object that was given by the user.\n\nResolve these two problems by introducing a flag NOT_USER_GIVEN that\nmarks blobs and trees that are non-user-given, replacing USER_GIVEN.\n(Only blobs and trees are marked because this mark is only used when\nfiltering objects, and filtering of other types of objects is not\nsupported yet.)\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 31 ++++++++++++++++++-------------\n revision.c | 1 -\n revision.h | 10 +++++++---\n 3 files changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 243192af5..7a1a0929d 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -53,7 +53,7 @@ static void process_blob(struct traversal_context *ctx,\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n \t\t\t\t path->buf, &path->buf[pathlen],\n \t\t\t\t ctx->filter_data);\n@@ -120,17 +120,19 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\tcontinue;\n \t\t}\n \n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\tstruct tree *t = lookup_tree(the_repository, entry.oid);\n+\t\t\tt->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_tree(ctx, t, base, entry.path);\n+\t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid->hash,\n \t\t\t\t\tbase, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\telse {\n+\t\t\tstruct blob *b = lookup_blob(the_repository, entry.oid);\n+\t\t\tb->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_blob(ctx, b, base, entry.path);\n+\t\t}\n \t}\n }\n \n@@ -171,7 +173,7 @@ static void process_tree(struct traversal_context *ctx,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -185,7 +187,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (!failed_parse)\n \t\tprocess_tree_contents(ctx, tree, base);\n \n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -301,8 +303,11 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n-\t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tif (get_commit_tree(commit)) {\n+\t\t\tstruct tree *tree = get_commit_tree(commit);\n+\t\t\ttree->object.flags |= NOT_USER_GIVEN;\n+\t\t\tadd_pending_tree(ctx->revs, tree);\n+\t\t}\n \t\tctx->show_commit(commit, ctx->show_data);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\ndiff --git a/revision.c b/revision.c\nindex 062749437..6d355b43c 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -175,7 +175,6 @@ static void add_pending_object_with_path(struct rev_info *revs,\n \t\tstrbuf_release(&buf);\n \t\treturn; /* do not add the commit itself */\n \t}\n-\tobj->flags |= USER_GIVEN;\n \tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n }\n \ndiff --git a/revision.h b/revision.h\nindex 5118aaaa9..2d381e636 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -8,7 +8,11 @@\n #include \"diff.h\"\n #include \"commit-slab-decl.h\"\n \n-/* Remember to update object flag allocation in object.h */\n+/* Remember to update object flag allocation in object.h\n+ * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n+ * filtering trees and blobs, but it may be useful to support filtering commits\n+ * in the future.\n+ */\n #define SEEN\t\t(1u<<0)\n #define UNINTERESTING (1u<<1)\n #define TREESAME\t(1u<<2)\n@@ -20,9 +24,9 @@\n #define SYMMETRIC_LEFT\t(1u<<8)\n #define PATCHSAME\t(1u<<9)\n #define BOTTOM\t\t(1u<<10)\n-#define USER_GIVEN\t(1u<<25) /* given directly by the user */\n+#define NOT_USER_GIVEN\t(1u<<25) /* tree or blob not given directly by user */\n #define TRACK_LINEAR\t(1u<<26)\n-#define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n+#define ALL_REV_FLAGS\t(((1u<<11)-1) | NOT_USER_GIVEN | TRACK_LINEAR)\n \n #define DECORATE_SHORT_REFS\t1\n #define DECORATE_FULL_REFS\t2\n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355785","messageId":"5d3b4e4acb73009e4cefecd0965fe5dd371efea1.1534374650.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1534374650.git.matvore@google.com","subject":"[PATCH v6 6/6] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-15T23:19:11Z","receivedAt":"2018-08-15T23:23:24Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Teach list-objects the \"tree:0\" filter which allows for filtering\nout all tree and blob objects (unless other objects are explicitly\nspecified by the user). The purpose of this patch is to allow smaller\npartial clones.\n\nThe name of this filter - tree:0 - does not explicitly specify that\nit also filters out all blobs, but this should not cause much confusion\nbecause blobs are not at all useful without the trees that refer to\nthem.\n\nI also consider only:commits as a name, but this is inaccurate because\nit suggests that annotated tags are omitted, but actually they are\nincluded.\n\nThe name \"tree:0\" allows later filtering based on depth, i.e. \"tree:1\"\nwould filter out all but the root tree and blobs. In order to avoid\nconfusion between 0 and capital O, the documentation was worded in a\nsomewhat round-about way that also hints at this future improvement to\nthe feature.\n\nSigned-off-by: Matthew DeVore \n---\n Documentation/rev-list-options.txt | 5 +++\n list-objects-filter-options.c | 4 +++\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 50 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 28 +++++++++++++++\n t/t5616-partial-clone.sh | 38 ++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 12 +++++++\n 7 files changed, 138 insertions(+)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 7b273635d..5f1672913 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -731,6 +731,11 @@ the requested refs.\n +\n The form '--filter=sparse:path=' similarly uses a sparse-checkout\n specification contained in .\n++\n+The form '--filter=tree:' omits all blobs and trees whose depth\n+from the root tree is >= (minimum depth if an object is located\n+at multiple depths in the commits traversed). Currently, only =0\n+is supported, which omits all blobs and trees.\n \n --no-filter::\n \tTurn off any previous `--filter=` argument.\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex c0e2bd6a0..a28382940 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -50,6 +50,10 @@ static int gently_parse_list_objects_filter(\n \t\t\treturn 0;\n \t\t}\n \n+\t} else if (!strcmp(arg, \"tree:0\")) {\n+\t\tfilter_options->choice = LOFC_TREE_NONE;\n+\t\treturn 0;\n+\n \t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n \t\tstruct object_context oc;\n \t\tstruct object_id sparse_oid;\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 0000a61f8..af64e5c66 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -10,6 +10,7 @@ enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n+\tLOFC_TREE_NONE,\n \tLOFC_SPARSE_OID,\n \tLOFC_SPARSE_PATH,\n \tLOFC__COUNT /* must be last */\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex a0ba78b20..8e3caf5bf 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -80,6 +80,55 @@ static void *filter_blobs_none__init(\n \treturn d;\n }\n \n+/*\n+ * A filter for list-objects to omit ALL trees and blobs from the traversal.\n+ * Can OPTIONALLY collect a list of the omitted OIDs.\n+ */\n+struct filter_trees_none_data {\n+\tstruct oidset *omits;\n+};\n+\n+static enum list_objects_filter_result filter_trees_none(\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_trees_none_data *filter_data = filter_data_;\n+\n+\tswitch (filter_situation) {\n+\tdefault:\n+\t\tdie(\"unknown filter_situation\");\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFS_BEGIN_TREE:\n+\tcase LOFS_BLOB:\n+\t\tif (filter_data->omits)\n+\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n+\n+\tcase LOFS_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\treturn LOFR_ZERO;\n+\n+\t}\n+}\n+\n+static void* filter_trees_none__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn)\n+{\n+\tstruct filter_trees_none_data *d = xcalloc(1, sizeof(*d));\n+\td->omits = omitted;\n+\n+\t*filter_fn = filter_trees_none;\n+\t*filter_free_fn = free;\n+\treturn d;\n+}\n+\n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n@@ -374,6 +423,7 @@ static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n+\tfilter_trees_none__init,\n \tfilter_sparse_oid__init,\n \tfilter_sparse_path__init,\n };\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 5e35f33bf..7a4d49ea1 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -72,6 +72,34 @@ test_expect_success 'get an error for missing tree object' '\n \tgrep -q \"bad tree object\" bad_tree\n '\n \n+test_expect_success 'setup for tests of tree:0' '\n+\tmkdir r1/subtree &&\n+\techo \"This is a file in a subtree\" >r1/subtree/file &&\n+\tgit -C r1 add subtree/file &&\n+\tgit -C r1 commit -m subtree\n+'\n+\n+test_expect_success 'verify tree:0 packfile has no blobs or trees' '\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\t! grep -E \"tree|blob\" objs\n+'\n+\n+test_expect_success 'grab tree directly when using tree:0' '\n+\t# We should get the tree specified directly but not its blobs or subtrees.\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD:\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\tawk \"/tree|blob/{print \\$1}\" objs >trees_and_blobs &&\n+\tgit -C r1 rev-parse HEAD: >expected &&\n+\ttest_cmp trees_and_blobs expected\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex bbbe7537d..8eeb85fbc 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -154,6 +154,44 @@ test_expect_success 'partial clone with transfer.fsckobjects=1 uses index-pack -\n \tgrep \"git index-pack.*--fsck-objects\" trace\n '\n \n+test_expect_success 'use fsck before and after manually fetching a missing subtree' '\n+\t# push new commit so server has a subtree\n+\tmkdir src/dir &&\n+\techo \"in dir\" >src/dir/file.txt &&\n+\tgit -C src add dir/file.txt &&\n+\tgit -C src commit -m \"file in dir\" &&\n+\tgit -C src push -u srv master &&\n+\tSUBTREE=$(git -C src rev-parse HEAD:dir) &&\n+\n+\trm -rf dst &&\n+\tgit clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n+\tgit -C dst fsck &&\n+\n+\t# Make sure we only have commits, and all trees and blobs are missing.\n+\tgit -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n+\tawk -f print_1.awk fetched_objects \\\n+\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n+\tsort fetched_types -u >unique_types.observed &&\n+\techo commit >unique_types.expected &&\n+\ttest_cmp unique_types.observed unique_types.expected &&\n+\n+\t# Auto-fetch a tree with cat-file.\n+\tgit -C dst cat-file -p $SUBTREE >tree_contents &&\n+\tgrep file.txt tree_contents &&\n+\n+\t# fsck still works after an auto-fetch of a tree.\n+\tgit -C dst fsck &&\n+\n+\t# Auto-fetch all remaining trees and blobs with --missing=error\n+\tgit -C dst rev-list master --missing=error --objects >fetched_objects &&\n+\ttest_line_count = 70 fetched_objects &&\n+\tawk -f print_1.awk fetched_objects \\\n+\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n+\tsort fetched_types -u >unique_types.observed &&\n+\tprintf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n+\ttest_cmp unique_types.observed unique_types.expected\n+'\n+\n test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' '\n \trm -rf src dst &&\n \tgit init src &&\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex fc0f92a16..30bf1c73e 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -213,6 +213,18 @@ test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for tre\n \ttest_line_count = 0 rev_list_err\n '\n \n+# Test tree:0 filter.\n+\n+test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n+\tgit -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=tree:0 \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed s/~// \\\n+\t\t| xargs -n1 git -C r3 cat-file -t \\\n+\t\t| sort -u >filtered_types &&\n+\tprintf \"blob\\ntree\\n\" > expected &&\n+\ttest_cmp filtered_types expected\n+'\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.18.0.865.gffc8e1a3cd6-goog\n\n"},{"id":"355949","messageId":"CAGZ79kaWcGbyc2S5gOCU7NdvT4fN46jq4xK9MvTLAFBGhyuo2A@mail.gmail.com","threadId":"49086","inReplyTo":"5d3b4e4acb73009e4cefecd0965fe5dd371efea1.1534374650.git.matvore@google.com","subject":"Re: [PATCH v6 6/6] list-objects-filter: implement filter tree:0","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-08-17T21:42:40Z","receivedAt":"2018-08-17T21:42:54Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Aug 15, 2018 at 4:23 PM Matthew DeVore wrote:\n>\n> Teach list-objects the \"tree:0\" filter which allows for filtering\n> out all tree and blob objects (unless other objects are explicitly\n> specified by the user). The purpose of this patch is to allow smaller\n> partial clones.\n>\n> The name of this filter - tree:0 - does not explicitly specify that\n> it also filters out all blobs, but this should not cause much confusion\n> because blobs are not at all useful without the trees that refer to\n> them.\n>\n> I also consider only:commits as a name, but this is inaccurate because\n> it suggests that annotated tags are omitted, but actually they are\n> included.\n\nSpeaking of tag objects, it is possible to tag anything, including blobs.\nWould a blob that is tagged (hence reachable without a tree) be not\nfiltered by tree:0 (or in the future any deeper depth) ?\n\nI found this series a good read, despite my unfamiliarity of the\npartial cloning.\n\nOne situation where I scratched my head for a second were previous patches\nthat use \"test_line_count = 0 rev_list_err\" whereas using test_must_be_empty\nwould be an equally good choice (I am more used to the latter than the former)\n\nThanks,\nStefan\n"},{"id":"355953","messageId":"CAMfpvhK3ouxa5H1ZWy_ZrQBjSq_qwavOtLe98SshaUmjTyBQXw@mail.gmail.com","threadId":"49086","inReplyTo":"CAGZ79kaWcGbyc2S5gOCU7NdvT4fN46jq4xK9MvTLAFBGhyuo2A@mail.gmail.com","subject":"Re: [PATCH v6 6/6] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-17T22:19:54Z","receivedAt":"2018-08-17T22:20:09Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Fri, Aug 17, 2018 at 2:42 PM Stefan Beller wrote:\n>\n> On Wed, Aug 15, 2018 at 4:23 PM Matthew DeVore wrote:\n> >\n> > Teach list-objects the \"tree:0\" filter which allows for filtering\n> > out all tree and blob objects (unless other objects are explicitly\n> > specified by the user). The purpose of this patch is to allow smaller\n> > partial clones.\n> >\n> > The name of this filter - tree:0 - does not explicitly specify that\n> > it also filters out all blobs, but this should not cause much confusion\n> > because blobs are not at all useful without the trees that refer to\n> > them.\n> >\n> > I also consider only:commits as a name, but this is inaccurate because\n> > it suggests that annotated tags are omitted, but actually they are\n> > included.\n>\n> Speaking of tag objects, it is possible to tag anything, including blobs.\n> Would a blob that is tagged (hence reachable without a tree) be not\n> filtered by tree:0 (or in the future any deeper depth) ?\nI think so. If I try to fetch a tagged tree or blob, it should fetch\nthat object itself, since I'm referring to it explicitly in the git\npack-objects arguments (I mention fetch since git rev-list apparently\ndoesn't support specifying non-commits on the command line). This is\nsimilar to how I can fetch a commit that would otherwise be filtered\n*if* I specify it explicitly (rather than a child commit).\n\nIf you're fetching a tagged tree, then for depth=0, it will fetch the\ngiven tree only, and not fetch any referents of an explicitly-given\ntree. For depth=1, it will fetch all direct referents.\n\nIf you're fetching a commit, then for depth=0, you will not get any\ntree objects, and for depth=1, you'll get only the root tree object\nand none of its referrents. So the commit itself is like a \"layer\" in\nthe depth count.\n\n>\n> I found this series a good read, despite my unfamiliarity of the\n> partial cloning.\n>\n> One situation where I scratched my head for a second were previous patches\n> that use \"test_line_count = 0 rev_list_err\" whereas using test_must_be_empty\n> would be an equally good choice (I am more used to the latter than the former)\n\nDone. Here is an interdiff (sorry, the tab characters are not\nmaintained by my mail client):\n\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex a1b93c72c..7e2f7ff26 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -200,14 +200,14 @@ test_expect_success 'missing tree objects with\n--missing=allow-promisor and --ex\n git -C repo config extensions.partialclone \"arbitrary string\" &&\n\n git -C repo rev-list --missing=allow-promisor --objects HEAD >objs\n2>rev_list_err &&\n- test_line_count = 0 rev_list_err &&\n+ test_must_be_empty rev_list_err &&\n # 3 commits, 3 blobs, and 1 tree\n test_line_count = 7 objs &&\n\n # Do the same for --exclude-promisor-objects, but with all trees gone.\n promise_and_delete $(git -C repo rev-parse baz^{tree}) &&\n git -C repo rev-list --exclude-promisor-objects --objects HEAD >objs\n2>rev_list_err &&\n- test_line_count = 0 rev_list_err &&\n+ test_must_be_empty rev_list_err &&\n # 3 commits, no blobs or trees\n test_line_count = 3 objs\n '\n@@ -226,7 +226,7 @@ test_expect_success 'missing non-root tree object\nand rev-list' '\n git -C repo config extensions.partialclone \"arbitrary string\" &&\n\n git -C repo rev-list --missing=allow-any --objects HEAD >objs\n2>rev_list_err &&\n- test_line_count = 0 rev_list_err &&\n+ test_must_be_empty rev_list_err &&\n # 1 commit and 1 tree\n test_line_count = 2 objs\n '\ndiff --git a/t/t6112-rev-list-filters-objects.sh\nb/t/t6112-rev-list-filters-objects.sh\nindex 30bf1c73e..27040d73a 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -206,11 +206,11 @@ test_expect_success 'rev-list W/ --missing=print\nand --missing=allow-any for tre\n test_cmp expected missing_objs &&\n\n # do not complain when a missing tree cannot be parsed\n- test_line_count = 0 rev_list_err &&\n+ test_must_be_empty rev_list_err &&\n\n git -C r3 rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n ! grep $TREE objs &&\n- test_line_count = 0 rev_list_err\n+ test_must_be_empty rev_list_err\n '\n\n # Test tree:0 filter.\n\n>\n> Thanks,\n> Stefan\n"},{"id":"355954","messageId":"CAGZ79kb3W6fNkqNemcdziXBB_cDmd3ngBU_0tdS_32rUhAW4dA@mail.gmail.com","threadId":"49086","inReplyTo":"CAMfpvhK3ouxa5H1ZWy_ZrQBjSq_qwavOtLe98SshaUmjTyBQXw@mail.gmail.com","subject":"Re: [PATCH v6 6/6] list-objects-filter: implement filter tree:0","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-08-17T22:28:11Z","receivedAt":"2018-08-17T22:28:25Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Aug 17, 2018 at 3:20 PM Matthew DeVore wrote:\n>\n> On Fri, Aug 17, 2018 at 2:42 PM Stefan Beller wrote:\n> >\n> > On Wed, Aug 15, 2018 at 4:23 PM Matthew DeVore wrote:\n> > >\n> > > Teach list-objects the \"tree:0\" filter which allows for filtering\n> > > out all tree and blob objects (unless other objects are explicitly\n> > > specified by the user). The purpose of this patch is to allow smaller\n> > > partial clones.\n> > >\n> > > The name of this filter - tree:0 - does not explicitly specify that\n> > > it also filters out all blobs, but this should not cause much confusion\n> > > because blobs are not at all useful without the trees that refer to\n> > > them.\n> > >\n> > > I also consider only:commits as a name, but this is inaccurate because\n> > > it suggests that annotated tags are omitted, but actually they are\n> > > included.\n> >\n> > Speaking of tag objects, it is possible to tag anything, including blobs.\n> > Would a blob that is tagged (hence reachable without a tree) be not\n> > filtered by tree:0 (or in the future any deeper depth) ?\n> I think so. If I try to fetch a tagged tree or blob, it should fetch\n> that object itself, since I'm referring to it explicitly in the git\n> pack-objects arguments (I mention fetch since git rev-list apparently\n> doesn't support specifying non-commits on the command line). This is\n> similar to how I can fetch a commit that would otherwise be filtered\n> *if* I specify it explicitly (rather than a child commit).\n>\n> If you're fetching a tagged tree, then for depth=0, it will fetch the\n> given tree only, and not fetch any referents of an explicitly-given\n> tree. For depth=1, it will fetch all direct referents.\n>\n> If you're fetching a commit, then for depth=0, you will not get any\n> tree objects, and for depth=1, you'll get only the root tree object\n> and none of its referrents. So the commit itself is like a \"layer\" in\n> the depth count.\n\nThat seems smart. Thanks!\n\n>\n> >\n> > I found this series a good read, despite my unfamiliarity of the\n> > partial cloning.\n> >\n> > One situation where I scratched my head for a second were previous patches\n> > that use \"test_line_count = 0 rev_list_err\" whereas using test_must_be_empty\n> > would be an equally good choice (I am more used to the latter than the former)\n>\n> Done. Here is an interdiff (sorry, the tab characters are not\n> maintained by my mail client):\n\nheh. Thanks for switching the style; I should have emphasized that\n(after reflection) I found them equally good, I am used to one\nover the other more.\n\nSo if that is the only issue brought up, I would not even ask for a resend.\n\nThanks,\nStefan\n"},{"id":"355999","messageId":"CACsJy8AE+MwBzzUFRGLKVp6vaAg2W_KO-qbUU2LQpd=rMQw2sA@mail.gmail.com","threadId":"49086","inReplyTo":"5d3b4e4acb73009e4cefecd0965fe5dd371efea1.1534374650.git.matvore@google.com","subject":"Re: [PATCH v6 6/6] list-objects-filter: implement filter tree:0","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-08-18T16:17:32Z","receivedAt":"2018-08-18T16:18:00Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Aug 16, 2018 at 1:54 AM Matthew DeVore wrote:\n> diff --git a/list-objects-filter.c b/list-objects-filter.c\n> index a0ba78b20..8e3caf5bf 100644\n> --- a/list-objects-filter.c\n> +++ b/list-objects-filter.c\n> @@ -80,6 +80,55 @@ static void *filter_blobs_none__init(\n> return d;\n> }\n>\n> +/*\n> + * A filter for list-objects to omit ALL trees and blobs from the traversal.\n> + * Can OPTIONALLY collect a list of the omitted OIDs.\n> + */\n> +struct filter_trees_none_data {\n> + struct oidset *omits;\n> +};\n> +\n> +static enum list_objects_filter_result filter_trees_none(\n> + enum list_objects_filter_situation filter_situation,\n> + struct object *obj,\n> + const char *pathname,\n> + const char *filename,\n> + void *filter_data_)\n> +{\n> + struct filter_trees_none_data *filter_data = filter_data_;\n> +\n> + switch (filter_situation) {\n> + default:\n> + die(\"unknown filter_situation\");\n\nThis sounds like BUG() than die() without _(). And you probably want\nto report filter_situation value too.\n\n> + return LOFR_ZERO;\n\nAnd neither BUG() or die() returns.\n-- \nDuy\n"},{"id":"356085","messageId":"msg.1534770125.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"Re: [PATCH v6 6/6] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-20T13:04:25Z","receivedAt":"2018-08-20T13:18:50Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"There were many instances in this file where it seemed like BUG would be\nbetter, so I created a new commit before this one to switch them over. The\ninterdiff is below.\n\nBTW, why are there so many instances of \"die\" without \"_\"? I expect all\nerrors that may be caused by a user to be localized.\n\nI'm going by the output of this: grep -IrE '\\Wdie\\([^_]' --exclude-dir=t\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 8e3caf5bf..09b2b05d5 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -44,8 +44,7 @@ static enum list_objects_filter_result filter_blobs_none(\n\n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -99,8 +98,7 @@ static enum list_objects_filter_result filter_trees_none(\n\n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n\n \tcase LOFS_BEGIN_TREE:\n \tcase LOFS_BLOB:\n@@ -151,8 +149,7 @@ static enum list_objects_filter_result \nfilter_blobs_limit(\n\n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -257,8 +254,7 @@ static enum list_objects_filter_result filter_sparse(\n\n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -439,7 +435,7 @@ void *list_objects_filter__init(\n \tassert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n\n \tif (filter_options->choice >= LOFC__COUNT)\n-\t\tdie(\"invalid list-objects filter choice: %d\",\n+\t\tBUG(\"invalid list-objects filter choice: %d\",\n \t\t filter_options->choice);\n\n \tinit_fn = s_filters[filter_options->choice];\n"},{"id":"356123","messageId":"CAGZ79kZfNeWnZEnX4Z0hMShxZZEsB2jYJ67JgE7F_d-3ymHi+A@mail.gmail.com","threadId":"49086","inReplyTo":"msg.1534770125.matvore@google.com","subject":"Re: [PATCH v6 6/6] list-objects-filter: implement filter tree:0","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-08-20T18:38:13Z","receivedAt":"2018-08-20T18:38:27Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Aug 20, 2018 at 6:18 AM Matthew DeVore wrote:\n>\n> There were many instances in this file where it seemed like BUG would be\n> better, so I created a new commit before this one to switch them over. The\n> interdiff is below.\n>\n> BTW, why are there so many instances of \"die\" without \"_\"? I expect all\n> errors that may be caused by a user to be localized.\n\nWell, there is the porcelain layer to be consumed by a human user\nand the plumbing that is good for scripts. And in scripts you might want\nto grep for certain errors and react to that, so a non-localized error\nmessage makes the script possible to run in any localisation.\n\nThe BUG is strictly for things that are due to Gits internals,\nnot for problematic user input. Problematic user input\ndefinitely wants a die(...), and depending on the plumbing/porcelain\nlayer it may need to be _(translatable).\n\nI think BUG() would never go with translated strings.\n\n> I'm going by the output of this: grep -IrE '\\Wdie\\([^_]' --exclude-dir=t\n>\n> diff --git a/list-objects-filter.c b/list-objects-filter.c\n> index 8e3caf5bf..09b2b05d5 100644\n> --- a/list-objects-filter.c\n> +++ b/list-objects-filter.c\n> @@ -44,8 +44,7 @@ static enum list_objects_filter_result filter_blobs_none(\n>\n> switch (filter_situation) {\n> default:\n> - die(\"unknown filter_situation\");\n> - return LOFR_ZERO;\n> + BUG(\"unknown filter_situation: %d\", filter_situation);\n>\n> case LOFS_BEGIN_TREE:\n> assert(obj->type == OBJ_TREE);\n> @@ -99,8 +98,7 @@ static enum list_objects_filter_result filter_trees_none(\n>\n> switch (filter_situation) {\n> default:\n> - die(\"unknown filter_situation\");\n> - return LOFR_ZERO;\n> + BUG(\"unknown filter_situation: %d\", filter_situation);\n>\n> case LOFS_BEGIN_TREE:\n> case LOFS_BLOB:\n> @@ -151,8 +149,7 @@ static enum list_objects_filter_result\n> filter_blobs_limit(\n>\n> switch (filter_situation) {\n> default:\n> - die(\"unknown filter_situation\");\n> - return LOFR_ZERO;\n> + BUG(\"unknown filter_situation: %d\", filter_situation);\n>\n> case LOFS_BEGIN_TREE:\n> assert(obj->type == OBJ_TREE);\n> @@ -257,8 +254,7 @@ static enum list_objects_filter_result filter_sparse(\n>\n> switch (filter_situation) {\n> default:\n> - die(\"unknown filter_situation\");\n> - return LOFR_ZERO;\n> + BUG(\"unknown filter_situation: %d\", filter_situation);\n\nUp until here we just have replace the die by BUG in the default\ncase of the state machine switch. (We need the default due to strict\ncompile flags, but as filter_situation is an enum I thought we would not\nas compilers are smart enough to see we got all values of the enum\ncovered).\n\nI agree that keeping the defaults and having a BUG() is reasonable.\n\n\n>\n> case LOFS_BEGIN_TREE:\n> assert(obj->type == OBJ_TREE);\n> @@ -439,7 +435,7 @@ void *list_objects_filter__init(\n> assert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n>\n> if (filter_options->choice >= LOFC__COUNT)\n> - die(\"invalid list-objects filter choice: %d\",\n> + BUG(\"invalid list-objects filter choice: %d\",\n> filter_options->choice);\n\nThis also makes sense, combined with the assert before, this looks like\nreally defensive code.\n\nI think this patch is a good idea!\n\nThanks,\nStefan\n"},{"id":"356148","messageId":"CAMfpvh+7NoOGAmypGSCAtxH7DxHTvVB8_C3OkWNC14+c3MBO4A@mail.gmail.com","threadId":"49086","inReplyTo":"CAGZ79kZfNeWnZEnX4Z0hMShxZZEsB2jYJ67JgE7F_d-3ymHi+A@mail.gmail.com","subject":"Re: [PATCH v6 6/6] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-20T23:20:50Z","receivedAt":"2018-08-20T23:21:05Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Mon, Aug 20, 2018 at 11:38 AM Stefan Beller wrote:\n>\n> On Mon, Aug 20, 2018 at 6:18 AM Matthew DeVore wrote:\n> >\n> > There were many instances in this file where it seemed like BUG would be\n> > better, so I created a new commit before this one to switch them over. The\n> > interdiff is below.\n> >\n> > BTW, why are there so many instances of \"die\" without \"_\"? I expect all\n> > errors that may be caused by a user to be localized.\n>\n> Well, there is the porcelain layer to be consumed by a human user\n> and the plumbing that is good for scripts. And in scripts you might want\n> to grep for certain errors and react to that, so a non-localized error\n> message makes the script possible to run in any localisation.\n>\n> The BUG is strictly for things that are due to Gits internals,\n> not for problematic user input. Problematic user input\n> definitely wants a die(...), and depending on the plumbing/porcelain\n> layer it may need to be _(translatable).\nAh I see. Plumbing commands are not translated. Makes perfect sense now.\n\n>\n> I think BUG() would never go with translated strings.\n>\n> > I'm going by the output of this: grep -IrE '\\Wdie\\([^_]' --exclude-dir=t\n> >\n> > diff --git a/list-objects-filter.c b/list-objects-filter.c\n> > index 8e3caf5bf..09b2b05d5 100644\n> > --- a/list-objects-filter.c\n> > +++ b/list-objects-filter.c\n> > @@ -44,8 +44,7 @@ static enum list_objects_filter_result filter_blobs_none(\n> >\n> > switch (filter_situation) {\n> > default:\n> > - die(\"unknown filter_situation\");\n> > - return LOFR_ZERO;\n> > + BUG(\"unknown filter_situation: %d\", filter_situation);\n> >\n> > case LOFS_BEGIN_TREE:\n> > assert(obj->type == OBJ_TREE);\n> > @@ -99,8 +98,7 @@ static enum list_objects_filter_result filter_trees_none(\n> >\n> > switch (filter_situation) {\n> > default:\n> > - die(\"unknown filter_situation\");\n> > - return LOFR_ZERO;\n> > + BUG(\"unknown filter_situation: %d\", filter_situation);\n> >\n> > case LOFS_BEGIN_TREE:\n> > case LOFS_BLOB:\n> > @@ -151,8 +149,7 @@ static enum list_objects_filter_result\n> > filter_blobs_limit(\n> >\n> > switch (filter_situation) {\n> > default:\n> > - die(\"unknown filter_situation\");\n> > - return LOFR_ZERO;\n> > + BUG(\"unknown filter_situation: %d\", filter_situation);\n> >\n> > case LOFS_BEGIN_TREE:\n> > assert(obj->type == OBJ_TREE);\n> > @@ -257,8 +254,7 @@ static enum list_objects_filter_result filter_sparse(\n> >\n> > switch (filter_situation) {\n> > default:\n> > - die(\"unknown filter_situation\");\n> > - return LOFR_ZERO;\n> > + BUG(\"unknown filter_situation: %d\", filter_situation);\n>\n> Up until here we just have replace the die by BUG in the default\n> case of the state machine switch. (We need the default due to strict\n> compile flags, but as filter_situation is an enum I thought we would not\n> as compilers are smart enough to see we got all values of the enum\n> covered).\nAt the risk of going on a tangent, I assumed this was because enums\nare really ints, and the \"default\" is there in case the enum somehow\ngot assigned to an int without a corresponding value. Either because\nof a cast from an int that was out-of-range, or new values that were\nobtained from arithmetic or bitwise operations on the declared enum\nvalues, which created undeclared values.\n\n>\n> I agree that keeping the defaults and having a BUG() is reasonable.\n>\n>\n> >\n> > case LOFS_BEGIN_TREE:\n> > assert(obj->type == OBJ_TREE);\n> > @@ -439,7 +435,7 @@ void *list_objects_filter__init(\n> > assert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n> >\n> > if (filter_options->choice >= LOFC__COUNT)\n> > - die(\"invalid list-objects filter choice: %d\",\n> > + BUG(\"invalid list-objects filter choice: %d\",\n> > filter_options->choice);\n>\n> This also makes sense, combined with the assert before, this looks like\n> really defensive code.\n>\n> I think this patch is a good idea!\nThank you for your feedback :)\n\n>\n> Thanks,\n> Stefan\n"},{"id":"356149","messageId":"CAMfpvhKPJijO_E2FRh-UtrOthEc2YTaG0=1ToYy3akjzWUZWiw@mail.gmail.com","threadId":"49086","inReplyTo":"CAGZ79kb3W6fNkqNemcdziXBB_cDmd3ngBU_0tdS_32rUhAW4dA@mail.gmail.com","subject":"Re: [PATCH v6 6/6] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-08-20T23:30:15Z","receivedAt":"2018-08-20T23:30:30Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Fri, Aug 17, 2018 at 3:28 PM Stefan Beller wrote:\n>\n> On Fri, Aug 17, 2018 at 3:20 PM Matthew DeVore wrote:\n> >\n> > On Fri, Aug 17, 2018 at 2:42 PM Stefan Beller wrote:\n> > >\n> > > On Wed, Aug 15, 2018 at 4:23 PM Matthew DeVore wrote:\n> > > >\n> > > > Teach list-objects the \"tree:0\" filter which allows for filtering\n> > > > out all tree and blob objects (unless other objects are explicitly\n> > > > specified by the user). The purpose of this patch is to allow smaller\n> > > > partial clones.\n> > > >\n> > > > The name of this filter - tree:0 - does not explicitly specify that\n> > > > it also filters out all blobs, but this should not cause much confusion\n> > > > because blobs are not at all useful without the trees that refer to\n> > > > them.\n> > > >\n> > > > I also consider only:commits as a name, but this is inaccurate because\n> > > > it suggests that annotated tags are omitted, but actually they are\n> > > > included.\n> > >\n> > > Speaking of tag objects, it is possible to tag anything, including blobs.\n> > > Would a blob that is tagged (hence reachable without a tree) be not\n> > > filtered by tree:0 (or in the future any deeper depth) ?\n> > I think so. If I try to fetch a tagged tree or blob, it should fetch\n> > that object itself, since I'm referring to it explicitly in the git\n> > pack-objects arguments (I mention fetch since git rev-list apparently\n> > doesn't support specifying non-commits on the command line). This is\n> > similar to how I can fetch a commit that would otherwise be filtered\n> > *if* I specify it explicitly (rather than a child commit).\n> >\n> > If you're fetching a tagged tree, then for depth=0, it will fetch the\n> > given tree only, and not fetch any referents of an explicitly-given\n> > tree. For depth=1, it will fetch all direct referents.\n> >\n> > If you're fetching a commit, then for depth=0, you will not get any\n> > tree objects, and for depth=1, you'll get only the root tree object\n> > and none of its referrents. So the commit itself is like a \"layer\" in\n> > the depth count.\n>\n> That seems smart. Thanks!\n>\n> >\n> > >\n> > > I found this series a good read, despite my unfamiliarity of the\n> > > partial cloning.\n> > >\n> > > One situation where I scratched my head for a second were previous patches\n> > > that use \"test_line_count = 0 rev_list_err\" whereas using test_must_be_empty\n> > > would be an equally good choice (I am more used to the latter than the former)\n> >\n> > Done. Here is an interdiff (sorry, the tab characters are not\n> > maintained by my mail client):\n>\n> heh. Thanks for switching the style; I should have emphasized that\n> (after reflection) I found them equally good, I am used to one\n> over the other more.\nIt seems marginally better to me. I also noticed a clean-up patch\ngoing by that aggressively switched to test_must_be_empty wherever\npossible: https://public-inbox.org/git/20180819215725.29001-1-szeder.dev@gmail.com/\n\nOTOH, if it were up to me I would have just gotten rid of\ntest_must_be_empty and used an existing function with the right\nargument, like `test_cmp /dev/null` - but using some form consistently\nis the most important, whatever it is.\n\n>\n> So if that is the only issue brought up, I would not even ask for a resend.\n>\n> Thanks,\n> Stefan\n"},{"id":"356152","messageId":"CAGZ79kZfZ2FWfgrvb82YxXP7zHjYDp0J_581ZHgG3un1phHxGg@mail.gmail.com","threadId":"49086","inReplyTo":"CAMfpvhKPJijO_E2FRh-UtrOthEc2YTaG0=1ToYy3akjzWUZWiw@mail.gmail.com","subject":"Re: [PATCH v6 6/6] list-objects-filter: implement filter tree:0","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-08-21T00:29:50Z","receivedAt":"2018-08-21T00:30:04Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> > heh. Thanks for switching the style; I should have emphasized that\n> > (after reflection) I found them equally good, I am used to one\n> > over the other more.\n> It seems marginally better to me. I also noticed a clean-up patch\n> going by that aggressively switched to test_must_be_empty wherever\n> possible: https://public-inbox.org/git/20180819215725.29001-1-szeder.dev@gmail.com/\n>\n> OTOH, if it were up to me I would have just gotten rid of\n> test_must_be_empty and used an existing function with the right\n> argument, like `test_cmp /dev/null` - but using some form consistently\n> is the most important, whatever it is.\n\n/dev/null, eh? It shows you don't use Windows on a day to day basis. ;-)\nBut yeah consistency is really good to have. :)\n"},{"id":"356153","messageId":"CAGZ79kaR7WARYFOGw8p=B4v9hPyZJ71529y-V0cnB-_tnLiEXw@mail.gmail.com","threadId":"49086","inReplyTo":"CAMfpvh+7NoOGAmypGSCAtxH7DxHTvVB8_C3OkWNC14+c3MBO4A@mail.gmail.com","subject":"Re: [PATCH v6 6/6] list-objects-filter: implement filter tree:0","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-08-21T00:36:15Z","receivedAt":"2018-08-21T00:36:29Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> At the risk of going on a tangent, I assumed this was because enums\n> are really ints, and the \"default\" is there in case the enum somehow\n> got assigned to an int without a corresponding value. Either because\n> of a cast from an int that was out-of-range, or new values that were\n> obtained from arithmetic or bitwise operations on the declared enum\n> values, which created undeclared values.\n\nSee\n374166cb381 (grep: catch a missing enum in switch statement, 2017-05-25)\nor a bit date, but nevertheless an interesting read:\nb8527d5fa61 (wt-status: fix possible use of uninitialized variable, 2013-03-21)\n\n...compilers these days are just too smart to reason about them :-)\n"},{"id":"356175","messageId":"CACsJy8CNyep5cEV2bWkAMiZNrd9uGi1i8He3jGhPFq4NtKAPzg@mail.gmail.com","threadId":"49086","inReplyTo":"CAGZ79kZfNeWnZEnX4Z0hMShxZZEsB2jYJ67JgE7F_d-3ymHi+A@mail.gmail.com","subject":"Re: [PATCH v6 6/6] list-objects-filter: implement filter tree:0","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-08-21T15:50:51Z","receivedAt":"2018-08-21T15:51:19Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Aug 20, 2018 at 8:38 PM Stefan Beller wrote:\n>\n> On Mon, Aug 20, 2018 at 6:18 AM Matthew DeVore wrote:\n> >\n> > There were many instances in this file where it seemed like BUG would be\n> > better, so I created a new commit before this one to switch them over. The\n> > interdiff is below.\n> >\n> > BTW, why are there so many instances of \"die\" without \"_\"? I expect all\n> > errors that may be caused by a user to be localized.\n>\n> Well, there is the porcelain layer to be consumed by a human user\n> and the plumbing that is good for scripts. And in scripts you might want\n> to grep for certain errors and react to that, so a non-localized error\n> message makes the script possible to run in any localisation.\n\nI probably have a different view about this, but strings (as English\nsentences) are for human only and should be translated. For machines\nthere should be well defined format (that just might look like\nEnglish), not totally free text. In some case, this format can be as\nsimple as the \"error/warning/fatal\" prefix, which is left\nuntranslated, but the rest should be. There is no guarantee that these\ndie() messages will not change in the future, even left untranslated.\n-- \nDuy\n"},{"id":"356223","messageId":"xmqqlg8z75qo.fsf@gitster-ct.c.googlers.com","threadId":"49086","inReplyTo":"CAGZ79kZfZ2FWfgrvb82YxXP7zHjYDp0J_581ZHgG3un1phHxGg@mail.gmail.com","subject":"Re: [PATCH v6 6/6] list-objects-filter: implement filter tree:0","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-08-21T21:46:23Z","receivedAt":"2018-08-21T21:46:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n>> ...\n>> OTOH, if it were up to me I would have just gotten rid of\n>> test_must_be_empty and used an existing function with the right\n>> argument, like `test_cmp /dev/null` - but using some form consistently\n>> is the most important, whatever it is.\n>\n> /dev/null, eh? It shows you don't use Windows on a day to day basis. ;-)\n> But yeah consistency is really good to have. :)\n\nJust to make sure we don't give wrong impression to bystanders, do\nyou mean that we should discourage using /dev/null in our tests or\nscripts due to portability concerns?\n\nI thought they had good enough emulation that writing /dev/null on\nthe command line in scripts do what we expect the shell to do; the\nsame thing can be said for calling open(2) on \"/dev/null\".\n\nBack to the topic from the tangent, but there was a discussion on\nchoosing between \"test_must_be_empty actual\" vs \"test_cmp empty\nactual\", and was even a proposal to trigger an error when an empty\nfile is given to test_cmp. You two might want to join the party\nthere, perhaps?\n\n"},{"id":"356292","messageId":"CAGZ79kaxbgbroqQcih4yCWg=SQvUjuxwMMCTLSGv66rm7pes1w@mail.gmail.com","threadId":"49086","inReplyTo":"xmqqlg8z75qo.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v6 6/6] list-objects-filter: implement filter tree:0","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-08-22T18:00:29Z","receivedAt":"2018-08-22T18:00:44Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Aug 21, 2018 at 2:46 PM Junio C Hamano wrote:\n>\n> Stefan Beller writes:\n>\n> >> ...\n> >> OTOH, if it were up to me I would have just gotten rid of\n> >> test_must_be_empty and used an existing function with the right\n> >> argument, like `test_cmp /dev/null` - but using some form consistently\n> >> is the most important, whatever it is.\n> >\n> > /dev/null, eh? It shows you don't use Windows on a day to day basis. ;-)\n> > But yeah consistency is really good to have. :)\n>\n> Just to make sure we don't give wrong impression to bystanders, do\n> you mean that we should discourage using /dev/null in our tests or\n> scripts due to portability concerns?\n\nI would discourage reading /dev/null (as in `test_cmp /dev/null actual`)\nover a more specific `test_must_be_empty` as that is easier to read.\n(But I do neither en- or discourage the use of /dev/null in the implementation\nof that function).\n\n> I thought they had good enough emulation that writing /dev/null on\n> the command line in scripts do what we expect the shell to do; the\n> same thing can be said for calling open(2) on \"/dev/null\".\n\nOh, opening and reading is new to me, thanks!\n\n> Back to the topic from the tangent, but there was a discussion on\n> choosing between \"test_must_be_empty actual\" vs \"test_cmp empty\n> actual\", and was even a proposal to trigger an error when an empty\n> file is given to test_cmp.\n\nOh, that is an interesting way to ensure consistency.\n\n> You two might want to join the party\n> there, perhaps?\n\nI'll read into that.\n\nStefan\n"},{"id":"357314","messageId":"cover.1536081438.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"[PATCH v7 0/7] filter: support for excluding all trees and blobs","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-04T18:05:43Z","receivedAt":"2018-09-04T18:06:04Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"I made the following changes since v6 of the patchset:\n - (suggested by Duy Nguyen) add a new commit which replaces uses of die() with\n BUG() in list-objects-filter.c wherever it corresponds to a coding error.\n - Replace die() with BUG() in new code.\n - Replace test_line_count = 0 with test_must_be_empty in new tests since the\n trend seems to be, based on other RFCs in progress, that we are standardizing\n on that phraseology. See:\n https://public-inbox.org/git/20180819215725.29001-1-szeder.dev@gmail.com/\n\nAs asked in the last \"What's cooking in git.git\" post, the status of this patch\nis:\n - The original reviewer, Jonathan Tan, is on vacation and will be back later\n this week.\n - Stefan Beller has been reviewing the patchset in Jonathan's absence, and\n stated that it's a good read despite not being familiar with the code:\n https://public-inbox.org/git/CAGZ79kaWcGbyc2S5gOCU7NdvT4fN46jq4xK9MvTLAFBGhyuo2A@mail.gmail.com/\n - I haven't updated this patch in a while since we have been in RC for a while,\n but after this update I think it's ready. There hasn't been any comment or\n request for change to the patchset recently.\n\nMatthew DeVore (7):\n list-objects: store common func args in struct\n list-objects: refactor to process_tree_contents\n list-objects: always parse trees gently\n rev-list: handle missing tree objects properly\n revision: mark non-user-given objects instead\n list-objects-filter: use BUG rather than die\n list-objects-filter: implement filter tree:0\n\n Documentation/rev-list-options.txt | 5 +\n builtin/rev-list.c | 11 +-\n list-objects-filter-options.c | 4 +\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 60 ++++++-\n list-objects.c | 232 +++++++++++++------------\n revision.c | 1 -\n revision.h | 25 ++-\n t/t0410-partial-clone.sh | 45 +++++\n t/t5317-pack-objects-filter-objects.sh | 41 +++++\n t/t5616-partial-clone.sh | 38 ++++\n t/t6112-rev-list-filters-objects.sh | 29 ++++\n 12 files changed, 367 insertions(+), 125 deletions(-)\n\n-- \n2.19.0.rc1.350.ge57e33dbd1-goog\n\n"},{"id":"357315","messageId":"aaf4c2d93e25aae32275f8d58b04a44b599c6323.1536081438.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1536081438.git.matvore@google.com","subject":"[PATCH v7 1/7] list-objects: store common func args in struct","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-04T18:05:44Z","receivedAt":"2018-09-04T18:06:08Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will make utility functions easier to create, as done by the next\npatch.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 158 +++++++++++++++++++++++--------------------------\n 1 file changed, 74 insertions(+), 84 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex c99c47ac1..584518a3f 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -12,20 +12,25 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n \n-static void process_blob(struct rev_info *revs,\n+struct traversal_context {\n+\tstruct rev_info *revs;\n+\tshow_object_fn show_object;\n+\tshow_commit_fn show_commit;\n+\tvoid *show_data;\n+\tfilter_object_fn filter_fn;\n+\tvoid *filter_data;\n+};\n+\n+static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \n-\tif (!revs->blob_objects)\n+\tif (!ctx->revs->blob_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad blob object\");\n@@ -41,21 +46,21 @@ static void process_blob(struct rev_info *revs,\n \t * may cause the actual filter to report an incomplete list\n \t * of missing objects.\n \t */\n-\tif (revs->exclude_promisor_objects &&\n+\tif (ctx->revs->exclude_promisor_objects &&\n \t !has_object_file(&obj->oid) &&\n \t is_promisor_object(&obj->oid))\n \t\treturn;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BLOB, obj,\n-\t\t\t path->buf, &path->buf[pathlen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n+\t\t\t\t path->buf, &path->buf[pathlen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, path->buf, cb_data);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -81,26 +86,21 @@ static void process_blob(struct rev_info *revs,\n * the link, and how to do it. Whether it necessarily makes\n * any sense what-so-ever to ever do that is another issue.\n */\n-static void process_gitlink(struct rev_info *revs,\n+static void process_gitlink(struct traversal_context *ctx,\n \t\t\t const unsigned char *sha1,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data)\n+\t\t\t const char *name)\n {\n \t/* Nothing to do */\n }\n \n-static void process_tree(struct rev_info *revs,\n+static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &tree->object;\n+\tstruct rev_info *revs = ctx->revs;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n@@ -133,14 +133,14 @@ static void process_tree(struct rev_info *revs,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BEGIN_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, base->buf, cb_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -157,29 +157,25 @@ static void process_tree(struct rev_info *revs,\n \t\t}\n \n \t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(revs,\n+\t\t\tprocess_tree(ctx,\n \t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(revs, entry.oid->hash,\n-\t\t\t\t\tshow, base, entry.path,\n-\t\t\t\t\tcb_data);\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n \t\telse\n-\t\t\tprocess_blob(revs,\n+\t\t\tprocess_blob(ctx,\n \t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t}\n \n-\tif (!(obj->flags & USER_GIVEN) && filter_fn) {\n-\t\tr = filter_fn(LOFS_END_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \t\tif (r & LOFR_MARK_SEEN)\n \t\t\tobj->flags |= SEEN;\n \t\tif (r & LOFR_DO_SHOW)\n-\t\t\tshow(obj, base->buf, cb_data);\n+\t\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \t}\n \n \tstrbuf_setlen(base, baselen);\n@@ -242,19 +238,15 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n-static void traverse_trees_and_blobs(struct rev_info *revs,\n-\t\t\t\t struct strbuf *base,\n-\t\t\t\t show_object_fn show_object,\n-\t\t\t\t void *show_data,\n-\t\t\t\t filter_object_fn filter_fn,\n-\t\t\t\t void *filter_data)\n+static void traverse_trees_and_blobs(struct traversal_context *ctx,\n+\t\t\t\t struct strbuf *base)\n {\n \tint i;\n \n \tassert(base->len == 0);\n \n-\tfor (i = 0; i < revs->pending.nr; i++) {\n-\t\tstruct object_array_entry *pending = revs->pending.objects + i;\n+\tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n+\t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n@@ -262,62 +254,49 @@ static void traverse_trees_and_blobs(struct rev_info *revs,\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tshow_object(obj, name, show_data);\n+\t\t\tctx->show_object(obj, name, ctx->show_data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(revs, (struct tree *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(revs, (struct blob *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n \t\t oid_to_hex(&obj->oid), name);\n \t}\n-\tobject_array_clear(&revs->pending);\n+\tobject_array_clear(&ctx->revs->pending);\n }\n \n-static void do_traverse(struct rev_info *revs,\n-\t\t\tshow_commit_fn show_commit,\n-\t\t\tshow_object_fn show_object,\n-\t\t\tvoid *show_data,\n-\t\t\tfilter_object_fn filter_fn,\n-\t\t\tvoid *filter_data)\n+static void do_traverse(struct traversal_context *ctx)\n {\n \tstruct commit *commit;\n \tstruct strbuf csp; /* callee's scratch pad */\n \tstrbuf_init(&csp, PATH_MAX);\n \n-\twhile ((commit = get_revision(revs)) != NULL) {\n+\twhile ((commit = get_revision(ctx->revs)) != NULL) {\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n \t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(revs, get_commit_tree(commit));\n-\t\tshow_commit(commit, show_data);\n+\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tctx->show_commit(commit, ctx->show_data);\n \n-\t\tif (revs->tree_blobs_in_commit_order)\n+\t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n \t\t\t * NEEDSWORK: Adding the tree and then flushing it here\n \t\t\t * needs a reallocation for each commit. Can we pass the\n \t\t\t * tree directory without allocation churn?\n \t\t\t */\n-\t\t\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t\t\t show_object, show_data,\n-\t\t\t\t\t\t filter_fn, filter_data);\n+\t\t\ttraverse_trees_and_blobs(ctx, &csp);\n \t}\n-\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t show_object, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\ttraverse_trees_and_blobs(ctx, &csp);\n \tstrbuf_release(&csp);\n }\n \n@@ -326,7 +305,14 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t\t show_object_fn show_object,\n \t\t\t void *show_data)\n {\n-\tdo_traverse(revs, show_commit, show_object, show_data, NULL, NULL);\n+\tstruct traversal_context ctx;\n+\tctx.revs = revs;\n+\tctx.show_commit = show_commit;\n+\tctx.show_object = show_object;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\tctx.filter_data = NULL;\n+\tdo_traverse(&ctx);\n }\n \n void traverse_commit_list_filtered(\n@@ -337,14 +323,18 @@ void traverse_commit_list_filtered(\n \tvoid *show_data,\n \tstruct oidset *omitted)\n {\n-\tfilter_object_fn filter_fn = NULL;\n+\tstruct traversal_context ctx;\n \tfilter_free_fn filter_free_fn = NULL;\n-\tvoid *filter_data = NULL;\n-\n-\tfilter_data = list_objects_filter__init(omitted, filter_options,\n-\t\t\t\t\t\t&filter_fn, &filter_free_fn);\n-\tdo_traverse(revs, show_commit, show_object, show_data,\n-\t\t filter_fn, filter_data);\n-\tif (filter_data && filter_free_fn)\n-\t\tfilter_free_fn(filter_data);\n+\n+\tctx.revs = revs;\n+\tctx.show_object = show_object;\n+\tctx.show_commit = show_commit;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\n+\tctx.filter_data = list_objects_filter__init(omitted, filter_options,\n+\t\t\t\t\t\t &ctx.filter_fn, &filter_free_fn);\n+\tdo_traverse(&ctx);\n+\tif (ctx.filter_data && filter_free_fn)\n+\t\tfilter_free_fn(ctx.filter_data);\n }\n-- \n2.19.0.rc1.350.ge57e33dbd1-goog\n\n"},{"id":"357316","messageId":"f753322f09757bdafe62dae89fa53171956ed69e.1536081438.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1536081438.git.matvore@google.com","subject":"[PATCH v7 2/7] list-objects: refactor to process_tree_contents","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-04T18:05:45Z","receivedAt":"2018-09-04T18:06:10Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will be used in a follow-up patch to reduce indentation needed when\ninvoking the logic conditionally. i.e. rather than:\n\nif (foo) {\n\twhile (...) {\n\t\t/* this is very indented */\n\t}\n}\n\nwe will have:\n\nif (foo)\n\tprocess_tree_contents(...);\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 68 ++++++++++++++++++++++++++++++--------------------\n 1 file changed, 41 insertions(+), 27 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 584518a3f..ccc529e5e 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -94,6 +94,46 @@ static void process_gitlink(struct traversal_context *ctx,\n \t/* Nothing to do */\n }\n \n+static void process_tree(struct traversal_context *ctx,\n+\t\t\t struct tree *tree,\n+\t\t\t struct strbuf *base,\n+\t\t\t const char *name);\n+\n+static void process_tree_contents(struct traversal_context *ctx,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *base)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tenum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ?\n+\t\tall_entries_interesting : entry_not_interesting;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tif (match != all_entries_interesting) {\n+\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n+\t\t\t\t\t\t &ctx->revs->diffopt.pathspec);\n+\t\t\tif (match == all_entries_not_interesting)\n+\t\t\t\tbreak;\n+\t\t\tif (match == entry_not_interesting)\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (S_ISDIR(entry.mode))\n+\t\t\tprocess_tree(ctx,\n+\t\t\t\t lookup_tree(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t\telse if (S_ISGITLINK(entry.mode))\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash,\n+\t\t\t\t\tbase, entry.path);\n+\t\telse\n+\t\t\tprocess_blob(ctx,\n+\t\t\t\t lookup_blob(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t}\n+}\n+\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n@@ -101,10 +141,6 @@ static void process_tree(struct traversal_context *ctx,\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n-\tstruct tree_desc desc;\n-\tstruct name_entry entry;\n-\tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n-\t\tall_entries_interesting: entry_not_interesting;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \tint gently = revs->ignore_missing_links ||\n@@ -144,29 +180,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tinit_tree_desc(&desc, tree->buffer, tree->size);\n-\n-\twhile (tree_entry(&desc, &entry)) {\n-\t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n-\t\t\t\t\t\t &revs->diffopt.pathspec);\n-\t\t\tif (match == all_entries_not_interesting)\n-\t\t\t\tbreak;\n-\t\t\tif (match == entry_not_interesting)\n-\t\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t}\n+\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n-- \n2.19.0.rc1.350.ge57e33dbd1-goog\n\n"},{"id":"357317","messageId":"55da169659b3c04e3f38f876744ee76eb6d0db93.1536081438.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1536081438.git.matvore@google.com","subject":"[PATCH v7 3/7] list-objects: always parse trees gently","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-04T18:05:46Z","receivedAt":"2018-09-04T18:06:28Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"If parsing fails when revs->ignore_missing_links and\nrevs->exclude_promisor_objects are both false, we print the OID anyway\nin the die(\"bad tree object...\") call, so any message printed by\nparse_tree_gently() is superfluous.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex ccc529e5e..f9b51db7a 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,8 +143,6 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n-\tint gently = revs->ignore_missing_links ||\n-\t\t revs->exclude_promisor_objects;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -152,7 +150,7 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, gently) < 0) {\n+\tif (parse_tree_gently(tree, 1) < 0) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n-- \n2.19.0.rc1.350.ge57e33dbd1-goog\n\n"},{"id":"357318","messageId":"8b24c222d45cb3a45f55f04785d02cc617db63e3.1536081438.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1536081438.git.matvore@google.com","subject":"[PATCH v7 4/7] rev-list: handle missing tree objects properly","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-04T18:05:47Z","receivedAt":"2018-09-04T18:06:32Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Previously, we assumed only blob objects could be missing. This patch\nmakes rev-list handle missing trees like missing blobs. The --missing=*\nand --exclude-promisor-objects flags now work for trees as they already\ndo for blobs. This is demonstrated in t6112.\n\nSigned-off-by: Matthew DeVore \n---\n builtin/rev-list.c | 11 ++++---\n list-objects.c | 11 +++++--\n revision.h | 15 +++++++++\n t/t0410-partial-clone.sh | 45 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 13 ++++++++\n t/t6112-rev-list-filters-objects.sh | 17 ++++++++++\n 6 files changed, 105 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 5b07f3f4a..49d6deed7 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -6,6 +6,7 @@\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"object.h\"\n #include \"object-store.h\"\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n@@ -209,7 +210,8 @@ static inline void finish_object__ma(struct object *obj)\n \t */\n \tswitch (arg_missing_action) {\n \tcase MA_ERROR:\n-\t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n+\t\tdie(\"missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tcase MA_ALLOW_ANY:\n@@ -222,8 +224,8 @@ static inline void finish_object__ma(struct object *obj)\n \tcase MA_ALLOW_PROMISOR:\n \t\tif (is_promisor_object(&obj->oid))\n \t\t\treturn;\n-\t\tdie(\"unexpected missing blob object '%s'\",\n-\t\t oid_to_hex(&obj->oid));\n+\t\tdie(\"unexpected missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tdefault:\n@@ -235,7 +237,7 @@ static inline void finish_object__ma(struct object *obj)\n static int finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\tif (!has_object_file(&obj->oid)) {\n \t\tfinish_object__ma(obj);\n \t\treturn 1;\n \t}\n@@ -373,6 +375,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tinit_revisions(&revs, prefix);\n \trevs.abbrev = DEFAULT_ABBREV;\n \trevs.commit_format = CMIT_FMT_UNSPECIFIED;\n+\trevs.do_not_die_on_missing_tree = 1;\n \n \t/*\n \t * Scan the argument list before invoking setup_revisions(), so that we\ndiff --git a/list-objects.c b/list-objects.c\nindex f9b51db7a..243192af5 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,6 +143,7 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tint failed_parse;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -150,7 +151,9 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, 1) < 0) {\n+\n+\tfailed_parse = parse_tree_gently(tree, 1);\n+\tif (failed_parse) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n@@ -163,7 +166,8 @@ static void process_tree(struct traversal_context *ctx,\n \t\t is_promisor_object(&obj->oid))\n \t\t\treturn;\n \n-\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n+\t\tif (!revs->do_not_die_on_missing_tree)\n+\t\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \n \tstrbuf_addstr(base, name);\n@@ -178,7 +182,8 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tprocess_tree_contents(ctx, tree, base);\n+\tif (!failed_parse)\n+\t\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\ndiff --git a/revision.h b/revision.h\nindex c599c34da..5118aaaa9 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -125,6 +125,21 @@ struct rev_info {\n \t\t\tline_level_traverse:1,\n \t\t\ttree_blobs_in_commit_order:1,\n \n+\t\t\t/*\n+\t\t\t * Blobs are shown without regard for their existence.\n+\t\t\t * But not so for trees: unless exclude_promisor_objects\n+\t\t\t * is set and the tree in question is a promisor object;\n+\t\t\t * OR ignore_missing_links is set, the revision walker\n+\t\t\t * dies with a \"bad tree object HASH\" message when\n+\t\t\t * encountering a missing tree. For callers that can\n+\t\t\t * handle missing trees and want them to be filterable\n+\t\t\t * and showable, set this to true. The revision walker\n+\t\t\t * will filter and show such a missing tree as usual,\n+\t\t\t * but will not attempt to recurse into this tree\n+\t\t\t * object.\n+\t\t\t */\n+\t\t\tdo_not_die_on_missing_tree:1,\n+\n \t\t\t/* for internal use only */\n \t\t\texclude_promisor_objects:1;\n \ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 4984ca583..7e2f7ff26 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -186,6 +186,51 @@ test_expect_success 'rev-list stops traversal at missing and promised commit' '\n \t! grep $FOO out\n '\n \n+test_expect_success 'missing tree objects with --missing=allow-promisor and --exclude-promisor-objects' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo foo &&\n+\ttest_commit -C repo bar &&\n+\ttest_commit -C repo baz &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse bar^{tree}) &&\n+\tpromise_and_delete $(git -C repo rev-parse foo^{tree}) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-promisor --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 3 commits, 3 blobs, and 1 tree\n+\ttest_line_count = 7 objs &&\n+\n+\t# Do the same for --exclude-promisor-objects, but with all trees gone.\n+\tpromise_and_delete $(git -C repo rev-parse baz^{tree}) &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 3 commits, no blobs or trees\n+\ttest_line_count = 3 objs\n+'\n+\n+test_expect_success 'missing non-root tree object and rev-list' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\tmkdir repo/dir &&\n+\techo foo > repo/dir/foo &&\n+\tgit -C repo add dir/foo &&\n+\tgit -C repo commit -m \"commit dir/foo\" &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse HEAD:dir) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 1 commit and 1 tree\n+\ttest_line_count = 2 objs\n+'\n+\n test_expect_success 'rev-list stops traversal at missing and promised tree' '\n \trm -rf repo &&\n \ttest_create_repo repo &&\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 6710c8bc8..5e35f33bf 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -59,6 +59,19 @@ test_expect_success 'verify normal and blob:none packfiles have same commits/tre\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'get an error for missing tree object' '\n+\tgit init r5 &&\n+\techo foo > r5/foo &&\n+\tgit -C r5 add foo &&\n+\tgit -C r5 commit -m \"foo\" &&\n+\tdel=$(git -C r5 rev-parse HEAD^{tree} | sed \"s|..|&/|\") &&\n+\trm r5/.git/objects/$del &&\n+\ttest_must_fail git -C r5 pack-objects --rev --stdout 2>bad_tree <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgrep -q \"bad tree object\" bad_tree\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 0a37dd5f9..d3d07975f 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -196,6 +196,23 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for trees' '\n+\tTREE=$(git -C r3 rev-parse HEAD:dir1) &&\n+\n+\trm r3/.git/objects/$(echo $TREE | sed \"s|^..|&/|\") &&\n+\n+\tgit -C r3 rev-list --quiet --missing=print --objects HEAD >missing_objs 2>rev_list_err &&\n+\techo \"?$TREE\" >expected &&\n+\ttest_cmp expected missing_objs &&\n+\n+\t# do not complain when a missing tree cannot be parsed\n+\ttest_must_be_empty rev_list_err &&\n+\n+\tgit -C r3 rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n+\t! grep $TREE objs &&\n+\ttest_must_be_empty rev_list_err\n+'\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.19.0.rc1.350.ge57e33dbd1-goog\n\n"},{"id":"357319","messageId":"6ada2b97faba0ed8012ecd00477eac4e33eeb76f.1536081438.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1536081438.git.matvore@google.com","subject":"[PATCH v7 5/7] revision: mark non-user-given objects instead","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-04T18:05:48Z","receivedAt":"2018-09-04T18:06:34Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Currently, list-objects.c incorrectly treats all root trees of commits\nas USER_GIVEN. Also, it would be easier to mark objects that are\nnon-user-given instead of user-given, since the places in the code\nwhere we access an object through a reference are more obvious than\nthe places where we access an object that was given by the user.\n\nResolve these two problems by introducing a flag NOT_USER_GIVEN that\nmarks blobs and trees that are non-user-given, replacing USER_GIVEN.\n(Only blobs and trees are marked because this mark is only used when\nfiltering objects, and filtering of other types of objects is not\nsupported yet.)\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 31 ++++++++++++++++++-------------\n revision.c | 1 -\n revision.h | 10 +++++++---\n 3 files changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 243192af5..7a1a0929d 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -53,7 +53,7 @@ static void process_blob(struct traversal_context *ctx,\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n \t\t\t\t path->buf, &path->buf[pathlen],\n \t\t\t\t ctx->filter_data);\n@@ -120,17 +120,19 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\tcontinue;\n \t\t}\n \n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\tstruct tree *t = lookup_tree(the_repository, entry.oid);\n+\t\t\tt->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_tree(ctx, t, base, entry.path);\n+\t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid->hash,\n \t\t\t\t\tbase, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\telse {\n+\t\t\tstruct blob *b = lookup_blob(the_repository, entry.oid);\n+\t\t\tb->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_blob(ctx, b, base, entry.path);\n+\t\t}\n \t}\n }\n \n@@ -171,7 +173,7 @@ static void process_tree(struct traversal_context *ctx,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -185,7 +187,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (!failed_parse)\n \t\tprocess_tree_contents(ctx, tree, base);\n \n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -301,8 +303,11 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n-\t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tif (get_commit_tree(commit)) {\n+\t\t\tstruct tree *tree = get_commit_tree(commit);\n+\t\t\ttree->object.flags |= NOT_USER_GIVEN;\n+\t\t\tadd_pending_tree(ctx->revs, tree);\n+\t\t}\n \t\tctx->show_commit(commit, ctx->show_data);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\ndiff --git a/revision.c b/revision.c\nindex 062749437..6d355b43c 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -175,7 +175,6 @@ static void add_pending_object_with_path(struct rev_info *revs,\n \t\tstrbuf_release(&buf);\n \t\treturn; /* do not add the commit itself */\n \t}\n-\tobj->flags |= USER_GIVEN;\n \tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n }\n \ndiff --git a/revision.h b/revision.h\nindex 5118aaaa9..2d381e636 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -8,7 +8,11 @@\n #include \"diff.h\"\n #include \"commit-slab-decl.h\"\n \n-/* Remember to update object flag allocation in object.h */\n+/* Remember to update object flag allocation in object.h\n+ * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n+ * filtering trees and blobs, but it may be useful to support filtering commits\n+ * in the future.\n+ */\n #define SEEN\t\t(1u<<0)\n #define UNINTERESTING (1u<<1)\n #define TREESAME\t(1u<<2)\n@@ -20,9 +24,9 @@\n #define SYMMETRIC_LEFT\t(1u<<8)\n #define PATCHSAME\t(1u<<9)\n #define BOTTOM\t\t(1u<<10)\n-#define USER_GIVEN\t(1u<<25) /* given directly by the user */\n+#define NOT_USER_GIVEN\t(1u<<25) /* tree or blob not given directly by user */\n #define TRACK_LINEAR\t(1u<<26)\n-#define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n+#define ALL_REV_FLAGS\t(((1u<<11)-1) | NOT_USER_GIVEN | TRACK_LINEAR)\n \n #define DECORATE_SHORT_REFS\t1\n #define DECORATE_FULL_REFS\t2\n-- \n2.19.0.rc1.350.ge57e33dbd1-goog\n\n"},{"id":"357320","messageId":"956f0c704f5492680928f4dc9ce9411e7f80bba4.1536081438.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1536081438.git.matvore@google.com","subject":"[PATCH v7 7/7] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-04T18:05:50Z","receivedAt":"2018-09-04T18:06:40Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Teach list-objects the \"tree:0\" filter which allows for filtering\nout all tree and blob objects (unless other objects are explicitly\nspecified by the user). The purpose of this patch is to allow smaller\npartial clones.\n\nThe name of this filter - tree:0 - does not explicitly specify that\nit also filters out all blobs, but this should not cause much confusion\nbecause blobs are not at all useful without the trees that refer to\nthem.\n\nI also considered only:commits as a name, but this is inaccurate because\nit suggests that annotated tags are omitted, but actually they are\nincluded.\n\nThe name \"tree:0\" allows later filtering based on depth, i.e. \"tree:1\"\nwould filter out all but the root tree and blobs. In order to avoid\nconfusion between 0 and capital O, the documentation was worded in a\nsomewhat round-about way that also hints at this future improvement to\nthe feature.\n\nSigned-off-by: Matthew DeVore \n---\n Documentation/rev-list-options.txt | 5 +++\n list-objects-filter-options.c | 4 +++\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 49 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 28 +++++++++++++++\n t/t5616-partial-clone.sh | 38 ++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 12 +++++++\n 7 files changed, 137 insertions(+)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 7b273635d..5f1672913 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -731,6 +731,11 @@ the requested refs.\n +\n The form '--filter=sparse:path=' similarly uses a sparse-checkout\n specification contained in .\n++\n+The form '--filter=tree:' omits all blobs and trees whose depth\n+from the root tree is >= (minimum depth if an object is located\n+at multiple depths in the commits traversed). Currently, only =0\n+is supported, which omits all blobs and trees.\n \n --no-filter::\n \tTurn off any previous `--filter=` argument.\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex c0e2bd6a0..a28382940 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -50,6 +50,10 @@ static int gently_parse_list_objects_filter(\n \t\t\treturn 0;\n \t\t}\n \n+\t} else if (!strcmp(arg, \"tree:0\")) {\n+\t\tfilter_options->choice = LOFC_TREE_NONE;\n+\t\treturn 0;\n+\n \t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n \t\tstruct object_context oc;\n \t\tstruct object_id sparse_oid;\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 0000a61f8..af64e5c66 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -10,6 +10,7 @@ enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n+\tLOFC_TREE_NONE,\n \tLOFC_SPARSE_OID,\n \tLOFC_SPARSE_PATH,\n \tLOFC__COUNT /* must be last */\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 5f8b1a002..09b2b05d5 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -79,6 +79,54 @@ static void *filter_blobs_none__init(\n \treturn d;\n }\n \n+/*\n+ * A filter for list-objects to omit ALL trees and blobs from the traversal.\n+ * Can OPTIONALLY collect a list of the omitted OIDs.\n+ */\n+struct filter_trees_none_data {\n+\tstruct oidset *omits;\n+};\n+\n+static enum list_objects_filter_result filter_trees_none(\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_trees_none_data *filter_data = filter_data_;\n+\n+\tswitch (filter_situation) {\n+\tdefault:\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n+\n+\tcase LOFS_BEGIN_TREE:\n+\tcase LOFS_BLOB:\n+\t\tif (filter_data->omits)\n+\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n+\n+\tcase LOFS_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\treturn LOFR_ZERO;\n+\n+\t}\n+}\n+\n+static void* filter_trees_none__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn)\n+{\n+\tstruct filter_trees_none_data *d = xcalloc(1, sizeof(*d));\n+\td->omits = omitted;\n+\n+\t*filter_fn = filter_trees_none;\n+\t*filter_free_fn = free;\n+\treturn d;\n+}\n+\n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n@@ -371,6 +419,7 @@ static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n+\tfilter_trees_none__init,\n \tfilter_sparse_oid__init,\n \tfilter_sparse_path__init,\n };\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 5e35f33bf..7a4d49ea1 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -72,6 +72,34 @@ test_expect_success 'get an error for missing tree object' '\n \tgrep -q \"bad tree object\" bad_tree\n '\n \n+test_expect_success 'setup for tests of tree:0' '\n+\tmkdir r1/subtree &&\n+\techo \"This is a file in a subtree\" >r1/subtree/file &&\n+\tgit -C r1 add subtree/file &&\n+\tgit -C r1 commit -m subtree\n+'\n+\n+test_expect_success 'verify tree:0 packfile has no blobs or trees' '\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\t! grep -E \"tree|blob\" objs\n+'\n+\n+test_expect_success 'grab tree directly when using tree:0' '\n+\t# We should get the tree specified directly but not its blobs or subtrees.\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD:\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\tawk \"/tree|blob/{print \\$1}\" objs >trees_and_blobs &&\n+\tgit -C r1 rev-parse HEAD: >expected &&\n+\ttest_cmp trees_and_blobs expected\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex bbbe7537d..8eeb85fbc 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -154,6 +154,44 @@ test_expect_success 'partial clone with transfer.fsckobjects=1 uses index-pack -\n \tgrep \"git index-pack.*--fsck-objects\" trace\n '\n \n+test_expect_success 'use fsck before and after manually fetching a missing subtree' '\n+\t# push new commit so server has a subtree\n+\tmkdir src/dir &&\n+\techo \"in dir\" >src/dir/file.txt &&\n+\tgit -C src add dir/file.txt &&\n+\tgit -C src commit -m \"file in dir\" &&\n+\tgit -C src push -u srv master &&\n+\tSUBTREE=$(git -C src rev-parse HEAD:dir) &&\n+\n+\trm -rf dst &&\n+\tgit clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n+\tgit -C dst fsck &&\n+\n+\t# Make sure we only have commits, and all trees and blobs are missing.\n+\tgit -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n+\tawk -f print_1.awk fetched_objects \\\n+\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n+\tsort fetched_types -u >unique_types.observed &&\n+\techo commit >unique_types.expected &&\n+\ttest_cmp unique_types.observed unique_types.expected &&\n+\n+\t# Auto-fetch a tree with cat-file.\n+\tgit -C dst cat-file -p $SUBTREE >tree_contents &&\n+\tgrep file.txt tree_contents &&\n+\n+\t# fsck still works after an auto-fetch of a tree.\n+\tgit -C dst fsck &&\n+\n+\t# Auto-fetch all remaining trees and blobs with --missing=error\n+\tgit -C dst rev-list master --missing=error --objects >fetched_objects &&\n+\ttest_line_count = 70 fetched_objects &&\n+\tawk -f print_1.awk fetched_objects \\\n+\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n+\tsort fetched_types -u >unique_types.observed &&\n+\tprintf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n+\ttest_cmp unique_types.observed unique_types.expected\n+'\n+\n test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' '\n \trm -rf src dst &&\n \tgit init src &&\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex d3d07975f..27040d73a 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -213,6 +213,18 @@ test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for tre\n \ttest_must_be_empty rev_list_err\n '\n \n+# Test tree:0 filter.\n+\n+test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n+\tgit -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=tree:0 \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed s/~// \\\n+\t\t| xargs -n1 git -C r3 cat-file -t \\\n+\t\t| sort -u >filtered_types &&\n+\tprintf \"blob\\ntree\\n\" > expected &&\n+\ttest_cmp filtered_types expected\n+'\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.19.0.rc1.350.ge57e33dbd1-goog\n\n"},{"id":"357321","messageId":"602ab6c3fa1e83139bd3131dc7e44c2a7e172d26.1536081438.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1536081438.git.matvore@google.com","subject":"[PATCH v7 6/7] list-objects-filter: use BUG rather than die","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-04T18:05:49Z","receivedAt":"2018-09-04T18:06:43Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"In some cases in this file, BUG makes more sense than die. In such\ncases, a we get there from a coding error rather than a user error.\n\n'return' has been removed following some instances of BUG since BUG does\nnot return.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter.c | 11 ++++-------\n 1 file changed, 4 insertions(+), 7 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex a0ba78b20..5f8b1a002 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -44,8 +44,7 @@ static enum list_objects_filter_result filter_blobs_none(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -102,8 +101,7 @@ static enum list_objects_filter_result filter_blobs_limit(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -208,8 +206,7 @@ static enum list_objects_filter_result filter_sparse(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -389,7 +386,7 @@ void *list_objects_filter__init(\n \tassert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n \n \tif (filter_options->choice >= LOFC__COUNT)\n-\t\tdie(\"invalid list-objects filter choice: %d\",\n+\t\tBUG(\"invalid list-objects filter choice: %d\",\n \t\t filter_options->choice);\n \n \tinit_fn = s_filters[filter_options->choice];\n-- \n2.19.0.rc1.350.ge57e33dbd1-goog\n\n"},{"id":"357323","messageId":"CAGZ79kbhXQPQ87kAw_bOd0L0k8W4q=nPX0ymLPNnaEnpKZyesw@mail.gmail.com","threadId":"49086","inReplyTo":"cover.1536081438.git.matvore@google.com","subject":"Re: [PATCH v7 0/7] filter: support for excluding all trees and blobs","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-09-04T18:41:49Z","receivedAt":"2018-09-04T18:42:05Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Sep 4, 2018 at 11:06 AM Matthew DeVore wrote:\n>\n> I made the following changes since v6 of the patchset:\n> - (suggested by Duy Nguyen) add a new commit which replaces uses of die() with\n> BUG() in list-objects-filter.c wherever it corresponds to a coding error.\n> - Replace die() with BUG() in new code.\n> - Replace test_line_count = 0 with test_must_be_empty in new tests since the\n> trend seems to be, based on other RFCs in progress, that we are standardizing\n> on that phraseology. See:\n> https://public-inbox.org/git/20180819215725.29001-1-szeder.dev@gmail.com/\n>\n> As asked in the last \"What's cooking in git.git\" post, the status of this patch\n> is:\n> - The original reviewer, Jonathan Tan, is on vacation and will be back later\n> this week.\n> - Stefan Beller has been reviewing the patchset in Jonathan's absence, and\n> stated that it's a good read despite not being familiar with the code:\n> https://public-inbox.org/git/CAGZ79kaWcGbyc2S5gOCU7NdvT4fN46jq4xK9MvTLAFBGhyuo2A@mail.gmail.com/\n\nand this still holds.\n\nThanks,\nStefan\n"},{"id":"357345","messageId":"xmqq5zzl80nw.fsf@gitster-ct.c.googlers.com","threadId":"49086","inReplyTo":"6ada2b97faba0ed8012ecd00477eac4e33eeb76f.1536081438.git.matvore@google.com","subject":"Re: [PATCH v7 5/7] revision: mark non-user-given objects instead","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-04T20:31:31Z","receivedAt":"2018-09-04T20:31:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthew DeVore writes:\n\n> diff --git a/revision.h b/revision.h\n> index 5118aaaa9..2d381e636 100644\n> --- a/revision.h\n> +++ b/revision.h\n> @@ -8,7 +8,11 @@\n> #include \"diff.h\"\n> #include \"commit-slab-decl.h\"\n> \n> -/* Remember to update object flag allocation in object.h */\n> +/* Remember to update object flag allocation in object.h\n> + * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n> + * filtering trees and blobs, but it may be useful to support filtering commits\n> + * in the future.\n> + */\n\nJust a minor style nit, but our multi-line comment begins with the\nopening \"/*\" (and closing \"*/\", too, but you got that right) on its\nown line, i.e.\n\n\t/*\n\t * Remember to update ...\n\n> -#define USER_GIVEN\t(1u<<25) /* given directly by the user */\n> +#define NOT_USER_GIVEN\t(1u<<25) /* tree or blob not given directly by user */\n\nIs \"given directly by user\" equivalent to \"given on the command\nline\"? Do objects given via \"--stdin\" count the same way? How abot\nthose given via \"--branches\" or \"A^@\"? Does \"not given directly by\nuser\" mean roughly the same thing as \"discovered by traversal\"?\n\nNot a suggestion to change anything in this patch, but if you can\ncome up with a better phrase that helps new readers' understanding\nso that they do not have to ask a question like this, that would be\ngreat.\n\nThanks.\n\n"},{"id":"357346","messageId":"xmqq1sa980mx.fsf@gitster-ct.c.googlers.com","threadId":"49086","inReplyTo":"602ab6c3fa1e83139bd3131dc7e44c2a7e172d26.1536081438.git.matvore@google.com","subject":"Re: [PATCH v7 6/7] list-objects-filter: use BUG rather than die","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-04T20:32:06Z","receivedAt":"2018-09-04T20:32:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthew DeVore writes:\n\n> In some cases in this file, BUG makes more sense than die. In such\n> cases, a we get there from a coding error rather than a user error.\n>\n> 'return' has been removed following some instances of BUG since BUG does\n> not return.\n>\n> Signed-off-by: Matthew DeVore \n> ---\n\nMakes sense.\n\n> list-objects-filter.c | 11 ++++-------\n> 1 file changed, 4 insertions(+), 7 deletions(-)\n>\n> diff --git a/list-objects-filter.c b/list-objects-filter.c\n> index a0ba78b20..5f8b1a002 100644\n> --- a/list-objects-filter.c\n> +++ b/list-objects-filter.c\n> @@ -44,8 +44,7 @@ static enum list_objects_filter_result filter_blobs_none(\n> \n> \tswitch (filter_situation) {\n> \tdefault:\n> -\t\tdie(\"unknown filter_situation\");\n> -\t\treturn LOFR_ZERO;\n> +\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n> \n> \tcase LOFS_BEGIN_TREE:\n> \t\tassert(obj->type == OBJ_TREE);\n> @@ -102,8 +101,7 @@ static enum list_objects_filter_result filter_blobs_limit(\n> \n> \tswitch (filter_situation) {\n> \tdefault:\n> -\t\tdie(\"unknown filter_situation\");\n> -\t\treturn LOFR_ZERO;\n> +\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n> \n> \tcase LOFS_BEGIN_TREE:\n> \t\tassert(obj->type == OBJ_TREE);\n> @@ -208,8 +206,7 @@ static enum list_objects_filter_result filter_sparse(\n> \n> \tswitch (filter_situation) {\n> \tdefault:\n> -\t\tdie(\"unknown filter_situation\");\n> -\t\treturn LOFR_ZERO;\n> +\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n> \n> \tcase LOFS_BEGIN_TREE:\n> \t\tassert(obj->type == OBJ_TREE);\n> @@ -389,7 +386,7 @@ void *list_objects_filter__init(\n> \tassert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n> \n> \tif (filter_options->choice >= LOFC__COUNT)\n> -\t\tdie(\"invalid list-objects filter choice: %d\",\n> +\t\tBUG(\"invalid list-objects filter choice: %d\",\n> \t\t filter_options->choice);\n> \n> \tinit_fn = s_filters[filter_options->choice];\n"},{"id":"357347","messageId":"xmqqwos16li9.fsf@gitster-ct.c.googlers.com","threadId":"49086","inReplyTo":"956f0c704f5492680928f4dc9ce9411e7f80bba4.1536081438.git.matvore@google.com","subject":"Re: [PATCH v7 7/7] list-objects-filter: implement filter tree:0","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-04T20:44:14Z","receivedAt":"2018-09-04T20:44:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthew DeVore writes:\n\n> @@ -50,6 +50,10 @@ static int gently_parse_list_objects_filter(\n> \t\t\treturn 0;\n> \t\t}\n> \n> +\t} else if (!strcmp(arg, \"tree:0\")) {\n> +\t\tfilter_options->choice = LOFC_TREE_NONE;\n> +\t\treturn 0;\n> +\n\nThis is not wrong per-se, but I would have expected to see something\nlike:\n\n\t... else if (skip_prefix(arg, \"tree:\", ¶m)) {\n\t\tunsigned long depth;\n\t\tif (!git_parse_ulong(param, &depth) || depth != 0) {\n\t\t\terr = \"only 'tree:0' is supported\";\n\t\t\treturn -1;\n\t\t}\n\t\tfilter_options->choice = LOFC_TREE_NONE;\n\t\treturn 0;\n\nso that \"tree:1\" is rejected not with \"invalid filter-spec\" but a\nbit more descriptive \"only tree:0 is\". Accepting \"tree:00\" or\n\"tree:0k\" is merely an added bogus^wbonus.\n\n"},{"id":"357459","messageId":"CAMfpvhJ-ncRMK8v+zL4bytpWhQvgYqDbpa5q-Tzc_qMYN2_j1w@mail.gmail.com","threadId":"49086","inReplyTo":"xmqq5zzl80nw.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v7 5/7] revision: mark non-user-given objects instead","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-05T18:00:15Z","receivedAt":"2018-09-05T18:00:32Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Tue, Sep 4, 2018 at 1:31 PM Junio C Hamano wrote:\n>\n> Matthew DeVore writes:\n>\n> > diff --git a/revision.h b/revision.h\n> > index 5118aaaa9..2d381e636 100644\n> > --- a/revision.h\n> > +++ b/revision.h\n> > @@ -8,7 +8,11 @@\n> > #include \"diff.h\"\n> > #include \"commit-slab-decl.h\"\n> >\n> > -/* Remember to update object flag allocation in object.h */\n> > +/* Remember to update object flag allocation in object.h\n> > + * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n> > + * filtering trees and blobs, but it may be useful to support filtering commits\n> > + * in the future.\n> > + */\n>\n> Just a minor style nit, but our multi-line comment begins with the\n> opening \"/*\" (and closing \"*/\", too, but you got that right) on its\n> own line, i.e.\n>\nFixed.\n\n> /*\n> * Remember to update ...\n>\n> > -#define USER_GIVEN (1u<<25) /* given directly by the user */\n> > +#define NOT_USER_GIVEN (1u<<25) /* tree or blob not given directly by user */\n>\n> Is \"given directly by user\" equivalent to \"given on the command\n> line\"? Do objects given via \"--stdin\" count the same way? How abot\n> those given via \"--branches\" or \"A^@\"? Does \"not given directly by\n> user\" mean roughly the same thing as \"discovered by traversal\"?\nNote that --branches and A^@ expands to commits, and commits can't be\nfiltered, so perhaps these questions will be unlikely for the time\nbeing. I did clarify the comment a bit.\n\nI also noticed that this commit fixes a bug - before this patch, \"git\nrev-list\" would potentially filter objects given on the command\nline/--stdin, while \"git pack-objects\" would not. That was because\neach command used a different function to populate the rev_info\nstruct, and rev-list's code path did not contain the hack which turned\non USER_GIVEN.\n\nSo I added a test to check the desired behavior. Here is an interdiff\nfor this commit only:\n\ndiff --git a/revision.h b/revision.h\nindex fe4ff1fec..83e164039 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -9,11 +9,7 @@\n #include \"diff.h\"\n #include \"commit-slab-decl.h\"\n\n-/* Remember to update object flag allocation in object.h\n- * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n- * filtering trees and blobs, but it may be useful to support filtering commits\n- * in the future.\n- */\n+/* Remember to update object flag allocation in object.h */\n #define SEEN (1u<<0)\n #define UNINTERESTING (1u<<1)\n #define TREESAME (1u<<2)\n@@ -25,7 +21,14 @@\n #define SYMMETRIC_LEFT (1u<<8)\n #define PATCHSAME (1u<<9)\n #define BOTTOM (1u<<10)\n-#define NOT_USER_GIVEN (1u<<25) /* tree or blob not given directly by user */\n+/*\n+ * Indicates object was reached by traversal. i.e. not given by user on\n+ * command-line or stdin.\n+ * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n+ * filtering trees and blobs, but it may be useful to support filtering commits\n+ * in the future.\n+ */\n+#define NOT_USER_GIVEN (1u<<25)\n #define TRACK_LINEAR (1u<<26)\n #define ALL_REV_FLAGS (((1u<<11)-1) | NOT_USER_GIVEN | TRACK_LINEAR)\n\ndiff --git a/t/t6112-rev-list-filters-objects.sh\nb/t/t6112-rev-list-filters-objects.sh\nindex 2e6a6a32e..a989a7082 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -30,6 +30,16 @@ test_expect_success 'verify blob:none omits all 5 blobs' '\n test_cmp observed expected\n '\n\n+test_expect_success 'specify blob explicitly prevents filtering' '\n+ file_3=$(git -C r1 ls-files -s file.3 \\\n+ | awk -f print_2.awk) &&\n+ file_4=$(git -C r1 ls-files -s file.4 \\\n+ | awk -f print_2.awk) &&\n+ git -C r1 rev-list HEAD --objects --filter=blob:none HEAD $file_3 >observed &&\n+ grep -q \"$file_3\" observed &&\n+ test_must_fail grep -q \"$file_4\" observed\n+'\n+\n test_expect_success 'verify emitted+omitted == all' '\n git -C r1 rev-list HEAD --objects \\\n | awk -f print_1.awk \\\n\n>\n> Not a suggestion to change anything in this patch, but if you can\n> come up with a better phrase that helps new readers' understanding\n> so that they do not have to ask a question like this, that would be\n> great.\n>\n> Thanks.\n>\n"},{"id":"357498","messageId":"CAMfpvh+m88Qb9mAPXuVMAhuUiUdzzEzFDLZ+ee6-OtxFHf4g_g@mail.gmail.com","threadId":"49086","inReplyTo":"xmqqwos16li9.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v7 7/7] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-06T00:08:37Z","receivedAt":"2018-09-06T00:08:53Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Tue, Sep 4, 2018 at 1:44 PM Junio C Hamano wrote:\n>\n> Matthew DeVore writes:\n>\n> > @@ -50,6 +50,10 @@ static int gently_parse_list_objects_filter(\n> > return 0;\n> > }\n> >\n> > + } else if (!strcmp(arg, \"tree:0\")) {\n> > + filter_options->choice = LOFC_TREE_NONE;\n> > + return 0;\n> > +\n>\n> This is not wrong per-se, but I would have expected to see something\n> like:\n>\n> ... else if (skip_prefix(arg, \"tree:\", ¶m)) {\n> unsigned long depth;\n> if (!git_parse_ulong(param, &depth) || depth != 0) {\n> err = \"only 'tree:0' is supported\";\n> return -1;\n> }\n> filter_options->choice = LOFC_TREE_NONE;\n> return 0;\n>\n> so that \"tree:1\" is rejected not with \"invalid filter-spec\" but a\n> bit more descriptive \"only tree:0 is\". Accepting \"tree:00\" or\n> \"tree:0k\" is merely an added bogus^wbonus.\n>\nGood idea. An interdiff for my fix is below. I didn't add a test,\nsince adding a shell test for every trivial error doesn't seem to\nscale, but let me know if you disagree. I did of course try provoking\nthe error manually.\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex a28382940..14f251de4 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -50,7 +50,17 @@ static int gently_parse_list_objects_filter(\n return 0;\n }\n\n- } else if (!strcmp(arg, \"tree:0\")) {\n+ } else if (skip_prefix(arg, \"tree:\", &v0)) {\n+ unsigned long depth;\n+ if (!git_parse_ulong(v0, &depth) || depth != 0) {\n+ if (errbuf) {\n+ strbuf_init(errbuf, 0);\n+ strbuf_addstr(\n+ errbuf,\n+ _(\"only 'tree:0' is supported\"));\n+ }\n+ return 1;\n+ }\n filter_options->choice = LOFC_TREE_NONE;\n return 0;\n"},{"id":"358103","messageId":"cover.1536885967.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"[PATCH v8 0/7] filter: support for excluding all trees and blobs","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-14T00:55:20Z","receivedAt":"2018-09-14T00:55:48Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Things seem to have settled down in terms of responses, so here is a re-roll,\nsome of the changes being Junio's suggestions:\n - show a more helpful error if a positive integer is given after \"tree:\"\n - added a test for an issue that this patchset inadvertently fixes:\n git rev-list would filter objects given explicitly on the command-line,\n but it should not.\n - improved documentation on the NOT_USER_GIVEN flag in revision.h\n\nMatthew DeVore (7):\n list-objects: store common func args in struct\n list-objects: refactor to process_tree_contents\n list-objects: always parse trees gently\n rev-list: handle missing tree objects properly\n revision: mark non-user-given objects instead\n list-objects-filter: use BUG rather than die\n list-objects-filter: implement filter tree:0\n\n Documentation/rev-list-options.txt | 5 +\n builtin/rev-list.c | 11 +-\n list-objects-filter-options.c | 14 ++\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 60 ++++++-\n list-objects.c | 232 +++++++++++++------------\n revision.c | 1 -\n revision.h | 26 ++-\n t/t0410-partial-clone.sh | 45 +++++\n t/t5317-pack-objects-filter-objects.sh | 41 +++++\n t/t5616-partial-clone.sh | 38 ++++\n t/t6112-rev-list-filters-objects.sh | 39 +++++\n 12 files changed, 389 insertions(+), 124 deletions(-)\n\n-- \n2.19.0.397.gdd90340f6a-goog\n\n"},{"id":"358104","messageId":"3de57eb7f9e662d93fd166b92c35719af9ea6c31.1536885967.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1536885967.git.matvore@google.com","subject":"[PATCH v8 1/7] list-objects: store common func args in struct","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-14T00:55:21Z","receivedAt":"2018-09-14T00:55:57Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will make utility functions easier to create, as done by the next\npatch.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 158 +++++++++++++++++++++++--------------------------\n 1 file changed, 74 insertions(+), 84 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex c99c47ac1..584518a3f 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -12,20 +12,25 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n \n-static void process_blob(struct rev_info *revs,\n+struct traversal_context {\n+\tstruct rev_info *revs;\n+\tshow_object_fn show_object;\n+\tshow_commit_fn show_commit;\n+\tvoid *show_data;\n+\tfilter_object_fn filter_fn;\n+\tvoid *filter_data;\n+};\n+\n+static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \n-\tif (!revs->blob_objects)\n+\tif (!ctx->revs->blob_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad blob object\");\n@@ -41,21 +46,21 @@ static void process_blob(struct rev_info *revs,\n \t * may cause the actual filter to report an incomplete list\n \t * of missing objects.\n \t */\n-\tif (revs->exclude_promisor_objects &&\n+\tif (ctx->revs->exclude_promisor_objects &&\n \t !has_object_file(&obj->oid) &&\n \t is_promisor_object(&obj->oid))\n \t\treturn;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BLOB, obj,\n-\t\t\t path->buf, &path->buf[pathlen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n+\t\t\t\t path->buf, &path->buf[pathlen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, path->buf, cb_data);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -81,26 +86,21 @@ static void process_blob(struct rev_info *revs,\n * the link, and how to do it. Whether it necessarily makes\n * any sense what-so-ever to ever do that is another issue.\n */\n-static void process_gitlink(struct rev_info *revs,\n+static void process_gitlink(struct traversal_context *ctx,\n \t\t\t const unsigned char *sha1,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data)\n+\t\t\t const char *name)\n {\n \t/* Nothing to do */\n }\n \n-static void process_tree(struct rev_info *revs,\n+static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &tree->object;\n+\tstruct rev_info *revs = ctx->revs;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n@@ -133,14 +133,14 @@ static void process_tree(struct rev_info *revs,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BEGIN_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, base->buf, cb_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -157,29 +157,25 @@ static void process_tree(struct rev_info *revs,\n \t\t}\n \n \t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(revs,\n+\t\t\tprocess_tree(ctx,\n \t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(revs, entry.oid->hash,\n-\t\t\t\t\tshow, base, entry.path,\n-\t\t\t\t\tcb_data);\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n \t\telse\n-\t\t\tprocess_blob(revs,\n+\t\t\tprocess_blob(ctx,\n \t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t}\n \n-\tif (!(obj->flags & USER_GIVEN) && filter_fn) {\n-\t\tr = filter_fn(LOFS_END_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \t\tif (r & LOFR_MARK_SEEN)\n \t\t\tobj->flags |= SEEN;\n \t\tif (r & LOFR_DO_SHOW)\n-\t\t\tshow(obj, base->buf, cb_data);\n+\t\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \t}\n \n \tstrbuf_setlen(base, baselen);\n@@ -242,19 +238,15 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n-static void traverse_trees_and_blobs(struct rev_info *revs,\n-\t\t\t\t struct strbuf *base,\n-\t\t\t\t show_object_fn show_object,\n-\t\t\t\t void *show_data,\n-\t\t\t\t filter_object_fn filter_fn,\n-\t\t\t\t void *filter_data)\n+static void traverse_trees_and_blobs(struct traversal_context *ctx,\n+\t\t\t\t struct strbuf *base)\n {\n \tint i;\n \n \tassert(base->len == 0);\n \n-\tfor (i = 0; i < revs->pending.nr; i++) {\n-\t\tstruct object_array_entry *pending = revs->pending.objects + i;\n+\tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n+\t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n@@ -262,62 +254,49 @@ static void traverse_trees_and_blobs(struct rev_info *revs,\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tshow_object(obj, name, show_data);\n+\t\t\tctx->show_object(obj, name, ctx->show_data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(revs, (struct tree *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(revs, (struct blob *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n \t\t oid_to_hex(&obj->oid), name);\n \t}\n-\tobject_array_clear(&revs->pending);\n+\tobject_array_clear(&ctx->revs->pending);\n }\n \n-static void do_traverse(struct rev_info *revs,\n-\t\t\tshow_commit_fn show_commit,\n-\t\t\tshow_object_fn show_object,\n-\t\t\tvoid *show_data,\n-\t\t\tfilter_object_fn filter_fn,\n-\t\t\tvoid *filter_data)\n+static void do_traverse(struct traversal_context *ctx)\n {\n \tstruct commit *commit;\n \tstruct strbuf csp; /* callee's scratch pad */\n \tstrbuf_init(&csp, PATH_MAX);\n \n-\twhile ((commit = get_revision(revs)) != NULL) {\n+\twhile ((commit = get_revision(ctx->revs)) != NULL) {\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n \t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(revs, get_commit_tree(commit));\n-\t\tshow_commit(commit, show_data);\n+\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tctx->show_commit(commit, ctx->show_data);\n \n-\t\tif (revs->tree_blobs_in_commit_order)\n+\t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n \t\t\t * NEEDSWORK: Adding the tree and then flushing it here\n \t\t\t * needs a reallocation for each commit. Can we pass the\n \t\t\t * tree directory without allocation churn?\n \t\t\t */\n-\t\t\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t\t\t show_object, show_data,\n-\t\t\t\t\t\t filter_fn, filter_data);\n+\t\t\ttraverse_trees_and_blobs(ctx, &csp);\n \t}\n-\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t show_object, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\ttraverse_trees_and_blobs(ctx, &csp);\n \tstrbuf_release(&csp);\n }\n \n@@ -326,7 +305,14 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t\t show_object_fn show_object,\n \t\t\t void *show_data)\n {\n-\tdo_traverse(revs, show_commit, show_object, show_data, NULL, NULL);\n+\tstruct traversal_context ctx;\n+\tctx.revs = revs;\n+\tctx.show_commit = show_commit;\n+\tctx.show_object = show_object;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\tctx.filter_data = NULL;\n+\tdo_traverse(&ctx);\n }\n \n void traverse_commit_list_filtered(\n@@ -337,14 +323,18 @@ void traverse_commit_list_filtered(\n \tvoid *show_data,\n \tstruct oidset *omitted)\n {\n-\tfilter_object_fn filter_fn = NULL;\n+\tstruct traversal_context ctx;\n \tfilter_free_fn filter_free_fn = NULL;\n-\tvoid *filter_data = NULL;\n-\n-\tfilter_data = list_objects_filter__init(omitted, filter_options,\n-\t\t\t\t\t\t&filter_fn, &filter_free_fn);\n-\tdo_traverse(revs, show_commit, show_object, show_data,\n-\t\t filter_fn, filter_data);\n-\tif (filter_data && filter_free_fn)\n-\t\tfilter_free_fn(filter_data);\n+\n+\tctx.revs = revs;\n+\tctx.show_object = show_object;\n+\tctx.show_commit = show_commit;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\n+\tctx.filter_data = list_objects_filter__init(omitted, filter_options,\n+\t\t\t\t\t\t &ctx.filter_fn, &filter_free_fn);\n+\tdo_traverse(&ctx);\n+\tif (ctx.filter_data && filter_free_fn)\n+\t\tfilter_free_fn(ctx.filter_data);\n }\n-- \n2.19.0.397.gdd90340f6a-goog\n\n"},{"id":"358105","messageId":"ecdcb996c9f26e612259ea028076df07e25b6c10.1536885967.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1536885967.git.matvore@google.com","subject":"[PATCH v8 2/7] list-objects: refactor to process_tree_contents","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-14T00:55:22Z","receivedAt":"2018-09-14T00:56:05Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will be used in a follow-up patch to reduce indentation needed when\ninvoking the logic conditionally. i.e. rather than:\n\nif (foo) {\n\twhile (...) {\n\t\t/* this is very indented */\n\t}\n}\n\nwe will have:\n\nif (foo)\n\tprocess_tree_contents(...);\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 68 ++++++++++++++++++++++++++++++--------------------\n 1 file changed, 41 insertions(+), 27 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 584518a3f..ccc529e5e 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -94,6 +94,46 @@ static void process_gitlink(struct traversal_context *ctx,\n \t/* Nothing to do */\n }\n \n+static void process_tree(struct traversal_context *ctx,\n+\t\t\t struct tree *tree,\n+\t\t\t struct strbuf *base,\n+\t\t\t const char *name);\n+\n+static void process_tree_contents(struct traversal_context *ctx,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *base)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tenum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ?\n+\t\tall_entries_interesting : entry_not_interesting;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tif (match != all_entries_interesting) {\n+\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n+\t\t\t\t\t\t &ctx->revs->diffopt.pathspec);\n+\t\t\tif (match == all_entries_not_interesting)\n+\t\t\t\tbreak;\n+\t\t\tif (match == entry_not_interesting)\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (S_ISDIR(entry.mode))\n+\t\t\tprocess_tree(ctx,\n+\t\t\t\t lookup_tree(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t\telse if (S_ISGITLINK(entry.mode))\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash,\n+\t\t\t\t\tbase, entry.path);\n+\t\telse\n+\t\t\tprocess_blob(ctx,\n+\t\t\t\t lookup_blob(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t}\n+}\n+\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n@@ -101,10 +141,6 @@ static void process_tree(struct traversal_context *ctx,\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n-\tstruct tree_desc desc;\n-\tstruct name_entry entry;\n-\tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n-\t\tall_entries_interesting: entry_not_interesting;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \tint gently = revs->ignore_missing_links ||\n@@ -144,29 +180,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tinit_tree_desc(&desc, tree->buffer, tree->size);\n-\n-\twhile (tree_entry(&desc, &entry)) {\n-\t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n-\t\t\t\t\t\t &revs->diffopt.pathspec);\n-\t\t\tif (match == all_entries_not_interesting)\n-\t\t\t\tbreak;\n-\t\t\tif (match == entry_not_interesting)\n-\t\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t}\n+\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n-- \n2.19.0.397.gdd90340f6a-goog\n\n"},{"id":"358106","messageId":"28f5ce6d48a69fd7326ce6a6995e5567a6f32d62.1536885967.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1536885967.git.matvore@google.com","subject":"[PATCH v8 3/7] list-objects: always parse trees gently","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-14T00:55:23Z","receivedAt":"2018-09-14T00:56:13Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"If parsing fails when revs->ignore_missing_links and\nrevs->exclude_promisor_objects are both false, we print the OID anyway\nin the die(\"bad tree object...\") call, so any message printed by\nparse_tree_gently() is superfluous.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex ccc529e5e..f9b51db7a 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,8 +143,6 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n-\tint gently = revs->ignore_missing_links ||\n-\t\t revs->exclude_promisor_objects;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -152,7 +150,7 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, gently) < 0) {\n+\tif (parse_tree_gently(tree, 1) < 0) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n-- \n2.19.0.397.gdd90340f6a-goog\n\n"},{"id":"358107","messageId":"9216afb6305ff2efd62ca056973d7548db9e00a0.1536885967.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1536885967.git.matvore@google.com","subject":"[PATCH v8 4/7] rev-list: handle missing tree objects properly","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-14T00:55:24Z","receivedAt":"2018-09-14T00:56:22Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Previously, we assumed only blob objects could be missing. This patch\nmakes rev-list handle missing trees like missing blobs. The --missing=*\nand --exclude-promisor-objects flags now work for trees as they already\ndo for blobs. This is demonstrated in t6112.\n\nSigned-off-by: Matthew DeVore \n---\n builtin/rev-list.c | 11 ++++---\n list-objects.c | 11 +++++--\n revision.h | 15 +++++++++\n t/t0410-partial-clone.sh | 45 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 13 ++++++++\n t/t6112-rev-list-filters-objects.sh | 17 ++++++++++\n 6 files changed, 105 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 5b07f3f4a..49d6deed7 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -6,6 +6,7 @@\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"object.h\"\n #include \"object-store.h\"\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n@@ -209,7 +210,8 @@ static inline void finish_object__ma(struct object *obj)\n \t */\n \tswitch (arg_missing_action) {\n \tcase MA_ERROR:\n-\t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n+\t\tdie(\"missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tcase MA_ALLOW_ANY:\n@@ -222,8 +224,8 @@ static inline void finish_object__ma(struct object *obj)\n \tcase MA_ALLOW_PROMISOR:\n \t\tif (is_promisor_object(&obj->oid))\n \t\t\treturn;\n-\t\tdie(\"unexpected missing blob object '%s'\",\n-\t\t oid_to_hex(&obj->oid));\n+\t\tdie(\"unexpected missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tdefault:\n@@ -235,7 +237,7 @@ static inline void finish_object__ma(struct object *obj)\n static int finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\tif (!has_object_file(&obj->oid)) {\n \t\tfinish_object__ma(obj);\n \t\treturn 1;\n \t}\n@@ -373,6 +375,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tinit_revisions(&revs, prefix);\n \trevs.abbrev = DEFAULT_ABBREV;\n \trevs.commit_format = CMIT_FMT_UNSPECIFIED;\n+\trevs.do_not_die_on_missing_tree = 1;\n \n \t/*\n \t * Scan the argument list before invoking setup_revisions(), so that we\ndiff --git a/list-objects.c b/list-objects.c\nindex f9b51db7a..243192af5 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,6 +143,7 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tint failed_parse;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -150,7 +151,9 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, 1) < 0) {\n+\n+\tfailed_parse = parse_tree_gently(tree, 1);\n+\tif (failed_parse) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n@@ -163,7 +166,8 @@ static void process_tree(struct traversal_context *ctx,\n \t\t is_promisor_object(&obj->oid))\n \t\t\treturn;\n \n-\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n+\t\tif (!revs->do_not_die_on_missing_tree)\n+\t\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \n \tstrbuf_addstr(base, name);\n@@ -178,7 +182,8 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tprocess_tree_contents(ctx, tree, base);\n+\tif (!failed_parse)\n+\t\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\ndiff --git a/revision.h b/revision.h\nindex 007278cc1..5910613cb 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -126,6 +126,21 @@ struct rev_info {\n \t\t\tline_level_traverse:1,\n \t\t\ttree_blobs_in_commit_order:1,\n \n+\t\t\t/*\n+\t\t\t * Blobs are shown without regard for their existence.\n+\t\t\t * But not so for trees: unless exclude_promisor_objects\n+\t\t\t * is set and the tree in question is a promisor object;\n+\t\t\t * OR ignore_missing_links is set, the revision walker\n+\t\t\t * dies with a \"bad tree object HASH\" message when\n+\t\t\t * encountering a missing tree. For callers that can\n+\t\t\t * handle missing trees and want them to be filterable\n+\t\t\t * and showable, set this to true. The revision walker\n+\t\t\t * will filter and show such a missing tree as usual,\n+\t\t\t * but will not attempt to recurse into this tree\n+\t\t\t * object.\n+\t\t\t */\n+\t\t\tdo_not_die_on_missing_tree:1,\n+\n \t\t\t/* for internal use only */\n \t\t\texclude_promisor_objects:1;\n \ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 128130066..f02b9ae37 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -186,6 +186,51 @@ test_expect_success 'rev-list stops traversal at missing and promised commit' '\n \t! grep $FOO out\n '\n \n+test_expect_success 'missing tree objects with --missing=allow-promisor and --exclude-promisor-objects' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo foo &&\n+\ttest_commit -C repo bar &&\n+\ttest_commit -C repo baz &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse bar^{tree}) &&\n+\tpromise_and_delete $(git -C repo rev-parse foo^{tree}) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-promisor --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 3 commits, 3 blobs, and 1 tree\n+\ttest_line_count = 7 objs &&\n+\n+\t# Do the same for --exclude-promisor-objects, but with all trees gone.\n+\tpromise_and_delete $(git -C repo rev-parse baz^{tree}) &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 3 commits, no blobs or trees\n+\ttest_line_count = 3 objs\n+'\n+\n+test_expect_success 'missing non-root tree object and rev-list' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\tmkdir repo/dir &&\n+\techo foo > repo/dir/foo &&\n+\tgit -C repo add dir/foo &&\n+\tgit -C repo commit -m \"commit dir/foo\" &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse HEAD:dir) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 1 commit and 1 tree\n+\ttest_line_count = 2 objs\n+'\n+\n test_expect_success 'rev-list stops traversal at missing and promised tree' '\n \trm -rf repo &&\n \ttest_create_repo repo &&\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 6710c8bc8..5e35f33bf 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -59,6 +59,19 @@ test_expect_success 'verify normal and blob:none packfiles have same commits/tre\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'get an error for missing tree object' '\n+\tgit init r5 &&\n+\techo foo > r5/foo &&\n+\tgit -C r5 add foo &&\n+\tgit -C r5 commit -m \"foo\" &&\n+\tdel=$(git -C r5 rev-parse HEAD^{tree} | sed \"s|..|&/|\") &&\n+\trm r5/.git/objects/$del &&\n+\ttest_must_fail git -C r5 pack-objects --rev --stdout 2>bad_tree <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgrep -q \"bad tree object\" bad_tree\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex d4ff0b3be..c662c97db 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -195,6 +195,23 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for trees' '\n+\tTREE=$(git -C r3 rev-parse HEAD:dir1) &&\n+\n+\trm r3/.git/objects/$(echo $TREE | sed \"s|^..|&/|\") &&\n+\n+\tgit -C r3 rev-list --quiet --missing=print --objects HEAD >missing_objs 2>rev_list_err &&\n+\techo \"?$TREE\" >expected &&\n+\ttest_cmp expected missing_objs &&\n+\n+\t# do not complain when a missing tree cannot be parsed\n+\ttest_must_be_empty rev_list_err &&\n+\n+\tgit -C r3 rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n+\t! grep $TREE objs &&\n+\ttest_must_be_empty rev_list_err\n+'\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.19.0.397.gdd90340f6a-goog\n\n"},{"id":"358108","messageId":"ca25f5914a98455cb6908a34146c10101d3529b0.1536885967.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1536885967.git.matvore@google.com","subject":"[PATCH v8 5/7] revision: mark non-user-given objects instead","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-14T00:55:25Z","receivedAt":"2018-09-14T00:56:30Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Currently, list-objects.c incorrectly treats all root trees of commits\nas USER_GIVEN. Also, it would be easier to mark objects that are\nnon-user-given instead of user-given, since the places in the code\nwhere we access an object through a reference are more obvious than\nthe places where we access an object that was given by the user.\n\nResolve these two problems by introducing a flag NOT_USER_GIVEN that\nmarks blobs and trees that are non-user-given, replacing USER_GIVEN.\n(Only blobs and trees are marked because this mark is only used when\nfiltering objects, and filtering of other types of objects is not\nsupported yet.)\n\nThis fixes a bug in that git rev-list behaved differently from git\npack-objects. pack-objects would *not* filter objects given explicitly\non the command line and rev-list would filter. This was because the two\ncommands used a different function to add objects to the rev_info\nstruct. This seems to have been an oversight, and pack-objects has the\ncorrect behavior, so I added a test to make sure that rev-list now\nbehaves properly.\n\nSigned-off-by: Matthew DeVore \n\nfixup of 6defd70de\n---\n list-objects.c | 31 +++++++++++++++++------------\n revision.c | 1 -\n revision.h | 11 ++++++++--\n t/t6112-rev-list-filters-objects.sh | 10 ++++++++++\n 4 files changed, 37 insertions(+), 16 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 243192af5..7a1a0929d 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -53,7 +53,7 @@ static void process_blob(struct traversal_context *ctx,\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n \t\t\t\t path->buf, &path->buf[pathlen],\n \t\t\t\t ctx->filter_data);\n@@ -120,17 +120,19 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\tcontinue;\n \t\t}\n \n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\tstruct tree *t = lookup_tree(the_repository, entry.oid);\n+\t\t\tt->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_tree(ctx, t, base, entry.path);\n+\t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid->hash,\n \t\t\t\t\tbase, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\telse {\n+\t\t\tstruct blob *b = lookup_blob(the_repository, entry.oid);\n+\t\t\tb->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_blob(ctx, b, base, entry.path);\n+\t\t}\n \t}\n }\n \n@@ -171,7 +173,7 @@ static void process_tree(struct traversal_context *ctx,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -185,7 +187,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (!failed_parse)\n \t\tprocess_tree_contents(ctx, tree, base);\n \n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -301,8 +303,11 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n-\t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tif (get_commit_tree(commit)) {\n+\t\t\tstruct tree *tree = get_commit_tree(commit);\n+\t\t\ttree->object.flags |= NOT_USER_GIVEN;\n+\t\t\tadd_pending_tree(ctx->revs, tree);\n+\t\t}\n \t\tctx->show_commit(commit, ctx->show_data);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\ndiff --git a/revision.c b/revision.c\nindex de4dce600..72d48a17f 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -175,7 +175,6 @@ static void add_pending_object_with_path(struct rev_info *revs,\n \t\tstrbuf_release(&buf);\n \t\treturn; /* do not add the commit itself */\n \t}\n-\tobj->flags |= USER_GIVEN;\n \tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n }\n \ndiff --git a/revision.h b/revision.h\nindex 5910613cb..83e164039 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -21,9 +21,16 @@\n #define SYMMETRIC_LEFT\t(1u<<8)\n #define PATCHSAME\t(1u<<9)\n #define BOTTOM\t\t(1u<<10)\n-#define USER_GIVEN\t(1u<<25) /* given directly by the user */\n+/*\n+ * Indicates object was reached by traversal. i.e. not given by user on\n+ * command-line or stdin.\n+ * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n+ * filtering trees and blobs, but it may be useful to support filtering commits\n+ * in the future.\n+ */\n+#define NOT_USER_GIVEN\t(1u<<25)\n #define TRACK_LINEAR\t(1u<<26)\n-#define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n+#define ALL_REV_FLAGS\t(((1u<<11)-1) | NOT_USER_GIVEN | TRACK_LINEAR)\n \n #define DECORATE_SHORT_REFS\t1\n #define DECORATE_FULL_REFS\t2\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex c662c97db..2e07dadf0 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -30,6 +30,16 @@ test_expect_success 'verify blob:none omits all 5 blobs' '\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'specify blob explicitly prevents filtering' '\n+\tfile_3=$(git -C r1 ls-files -s file.3 \\\n+\t\t| awk -f print_2.awk) &&\n+\tfile_4=$(git -C r1 ls-files -s file.4 \\\n+\t\t| awk -f print_2.awk) &&\n+\tgit -C r1 rev-list HEAD --objects --filter=blob:none HEAD $file_3 >observed &&\n+\tgrep -q \"$file_3\" observed &&\n+\ttest_must_fail grep -q \"$file_4\" observed\n+'\n+\n test_expect_success 'verify emitted+omitted == all' '\n \tgit -C r1 rev-list HEAD --objects \\\n \t\t| awk -f print_1.awk \\\n-- \n2.19.0.397.gdd90340f6a-goog\n\n"},{"id":"358109","messageId":"0e0e7306625e6e88b7779e900fd04acb365e2f78.1536885967.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1536885967.git.matvore@google.com","subject":"[PATCH v8 6/7] list-objects-filter: use BUG rather than die","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-14T00:55:26Z","receivedAt":"2018-09-14T00:56:38Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"In some cases in this file, BUG makes more sense than die. In such\ncases, a we get there from a coding error rather than a user error.\n\n'return' has been removed following some instances of BUG since BUG does\nnot return.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter.c | 11 ++++-------\n 1 file changed, 4 insertions(+), 7 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex a0ba78b20..5f8b1a002 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -44,8 +44,7 @@ static enum list_objects_filter_result filter_blobs_none(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -102,8 +101,7 @@ static enum list_objects_filter_result filter_blobs_limit(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -208,8 +206,7 @@ static enum list_objects_filter_result filter_sparse(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -389,7 +386,7 @@ void *list_objects_filter__init(\n \tassert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n \n \tif (filter_options->choice >= LOFC__COUNT)\n-\t\tdie(\"invalid list-objects filter choice: %d\",\n+\t\tBUG(\"invalid list-objects filter choice: %d\",\n \t\t filter_options->choice);\n \n \tinit_fn = s_filters[filter_options->choice];\n-- \n2.19.0.397.gdd90340f6a-goog\n\n"},{"id":"358110","messageId":"edba3791caf8bacc2f722f7874369f6776ecffe0.1536885967.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1536885967.git.matvore@google.com","subject":"[PATCH v8 7/7] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-14T00:55:27Z","receivedAt":"2018-09-14T00:56:46Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Teach list-objects the \"tree:0\" filter which allows for filtering\nout all tree and blob objects (unless other objects are explicitly\nspecified by the user). The purpose of this patch is to allow smaller\npartial clones.\n\nThe name of this filter - tree:0 - does not explicitly specify that\nit also filters out all blobs, but this should not cause much confusion\nbecause blobs are not at all useful without the trees that refer to\nthem.\n\nI also considered only:commits as a name, but this is inaccurate because\nit suggests that annotated tags are omitted, but actually they are\nincluded.\n\nThe name \"tree:0\" allows later filtering based on depth, i.e. \"tree:1\"\nwould filter out all but the root tree and blobs. In order to avoid\nconfusion between 0 and capital O, the documentation was worded in a\nsomewhat round-about way that also hints at this future improvement to\nthe feature.\n\nSigned-off-by: Matthew DeVore \n---\n Documentation/rev-list-options.txt | 5 +++\n list-objects-filter-options.c | 14 ++++++++\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 49 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 28 +++++++++++++++\n t/t5616-partial-clone.sh | 38 ++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 12 +++++++\n 7 files changed, 147 insertions(+)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 7b273635d..5f1672913 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -731,6 +731,11 @@ the requested refs.\n +\n The form '--filter=sparse:path=' similarly uses a sparse-checkout\n specification contained in .\n++\n+The form '--filter=tree:' omits all blobs and trees whose depth\n+from the root tree is >= (minimum depth if an object is located\n+at multiple depths in the commits traversed). Currently, only =0\n+is supported, which omits all blobs and trees.\n \n --no-filter::\n \tTurn off any previous `--filter=` argument.\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex c0e2bd6a0..14f251de4 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -50,6 +50,20 @@ static int gently_parse_list_objects_filter(\n \t\t\treturn 0;\n \t\t}\n \n+\t} else if (skip_prefix(arg, \"tree:\", &v0)) {\n+\t\tunsigned long depth;\n+\t\tif (!git_parse_ulong(v0, &depth) || depth != 0) {\n+\t\t\tif (errbuf) {\n+\t\t\t\tstrbuf_init(errbuf, 0);\n+\t\t\t\tstrbuf_addstr(\n+\t\t\t\t\terrbuf,\n+\t\t\t\t\t_(\"only 'tree:0' is supported\"));\n+\t\t\t}\n+\t\t\treturn 1;\n+\t\t}\n+\t\tfilter_options->choice = LOFC_TREE_NONE;\n+\t\treturn 0;\n+\n \t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n \t\tstruct object_context oc;\n \t\tstruct object_id sparse_oid;\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 0000a61f8..af64e5c66 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -10,6 +10,7 @@ enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n+\tLOFC_TREE_NONE,\n \tLOFC_SPARSE_OID,\n \tLOFC_SPARSE_PATH,\n \tLOFC__COUNT /* must be last */\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 5f8b1a002..09b2b05d5 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -79,6 +79,54 @@ static void *filter_blobs_none__init(\n \treturn d;\n }\n \n+/*\n+ * A filter for list-objects to omit ALL trees and blobs from the traversal.\n+ * Can OPTIONALLY collect a list of the omitted OIDs.\n+ */\n+struct filter_trees_none_data {\n+\tstruct oidset *omits;\n+};\n+\n+static enum list_objects_filter_result filter_trees_none(\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_trees_none_data *filter_data = filter_data_;\n+\n+\tswitch (filter_situation) {\n+\tdefault:\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n+\n+\tcase LOFS_BEGIN_TREE:\n+\tcase LOFS_BLOB:\n+\t\tif (filter_data->omits)\n+\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n+\n+\tcase LOFS_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\treturn LOFR_ZERO;\n+\n+\t}\n+}\n+\n+static void* filter_trees_none__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn)\n+{\n+\tstruct filter_trees_none_data *d = xcalloc(1, sizeof(*d));\n+\td->omits = omitted;\n+\n+\t*filter_fn = filter_trees_none;\n+\t*filter_free_fn = free;\n+\treturn d;\n+}\n+\n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n@@ -371,6 +419,7 @@ static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n+\tfilter_trees_none__init,\n \tfilter_sparse_oid__init,\n \tfilter_sparse_path__init,\n };\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 5e35f33bf..7a4d49ea1 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -72,6 +72,34 @@ test_expect_success 'get an error for missing tree object' '\n \tgrep -q \"bad tree object\" bad_tree\n '\n \n+test_expect_success 'setup for tests of tree:0' '\n+\tmkdir r1/subtree &&\n+\techo \"This is a file in a subtree\" >r1/subtree/file &&\n+\tgit -C r1 add subtree/file &&\n+\tgit -C r1 commit -m subtree\n+'\n+\n+test_expect_success 'verify tree:0 packfile has no blobs or trees' '\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\t! grep -E \"tree|blob\" objs\n+'\n+\n+test_expect_success 'grab tree directly when using tree:0' '\n+\t# We should get the tree specified directly but not its blobs or subtrees.\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD:\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\tawk \"/tree|blob/{print \\$1}\" objs >trees_and_blobs &&\n+\tgit -C r1 rev-parse HEAD: >expected &&\n+\ttest_cmp trees_and_blobs expected\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex bbbe7537d..8eeb85fbc 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -154,6 +154,44 @@ test_expect_success 'partial clone with transfer.fsckobjects=1 uses index-pack -\n \tgrep \"git index-pack.*--fsck-objects\" trace\n '\n \n+test_expect_success 'use fsck before and after manually fetching a missing subtree' '\n+\t# push new commit so server has a subtree\n+\tmkdir src/dir &&\n+\techo \"in dir\" >src/dir/file.txt &&\n+\tgit -C src add dir/file.txt &&\n+\tgit -C src commit -m \"file in dir\" &&\n+\tgit -C src push -u srv master &&\n+\tSUBTREE=$(git -C src rev-parse HEAD:dir) &&\n+\n+\trm -rf dst &&\n+\tgit clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n+\tgit -C dst fsck &&\n+\n+\t# Make sure we only have commits, and all trees and blobs are missing.\n+\tgit -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n+\tawk -f print_1.awk fetched_objects \\\n+\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n+\tsort fetched_types -u >unique_types.observed &&\n+\techo commit >unique_types.expected &&\n+\ttest_cmp unique_types.observed unique_types.expected &&\n+\n+\t# Auto-fetch a tree with cat-file.\n+\tgit -C dst cat-file -p $SUBTREE >tree_contents &&\n+\tgrep file.txt tree_contents &&\n+\n+\t# fsck still works after an auto-fetch of a tree.\n+\tgit -C dst fsck &&\n+\n+\t# Auto-fetch all remaining trees and blobs with --missing=error\n+\tgit -C dst rev-list master --missing=error --objects >fetched_objects &&\n+\ttest_line_count = 70 fetched_objects &&\n+\tawk -f print_1.awk fetched_objects \\\n+\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n+\tsort fetched_types -u >unique_types.observed &&\n+\tprintf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n+\ttest_cmp unique_types.observed unique_types.expected\n+'\n+\n test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' '\n \trm -rf src dst &&\n \tgit init src &&\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 2e07dadf0..a989a7082 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -222,6 +222,18 @@ test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for tre\n \ttest_must_be_empty rev_list_err\n '\n \n+# Test tree:0 filter.\n+\n+test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n+\tgit -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=tree:0 \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed s/~// \\\n+\t\t| xargs -n1 git -C r3 cat-file -t \\\n+\t\t| sort -u >filtered_types &&\n+\tprintf \"blob\\ntree\\n\" > expected &&\n+\ttest_cmp filtered_types expected\n+'\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.19.0.397.gdd90340f6a-goog\n\n"},{"id":"358137","messageId":"xmqqzhwkat8i.fsf@gitster-ct.c.googlers.com","threadId":"49086","inReplyTo":"ca25f5914a98455cb6908a34146c10101d3529b0.1536885967.git.matvore@google.com","subject":"Re: [PATCH v8 5/7] revision: mark non-user-given objects instead","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-14T17:23:09Z","receivedAt":"2018-09-14T17:23:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthew DeVore writes:\n\n> Currently, list-objects.c incorrectly treats all root trees of commits\n> as USER_GIVEN. Also, it would be easier to mark objects that are\n> non-user-given instead of user-given, since the places in the code\n> where we access an object through a reference are more obvious than\n> the places where we access an object that was given by the user.\n>\n> Resolve these two problems by introducing a flag NOT_USER_GIVEN that\n> marks blobs and trees that are non-user-given, replacing USER_GIVEN.\n> (Only blobs and trees are marked because this mark is only used when\n> filtering objects, and filtering of other types of objects is not\n> supported yet.)\n>\n> This fixes a bug in that git rev-list behaved differently from git\n> pack-objects. pack-objects would *not* filter objects given explicitly\n> on the command line and rev-list would filter. This was because the two\n> commands used a different function to add objects to the rev_info\n> struct. This seems to have been an oversight, and pack-objects has the\n> correct behavior, so I added a test to make sure that rev-list now\n> behaves properly.\n>\n> Signed-off-by: Matthew DeVore \n>\n> fixup of 6defd70de\n\nThat's probably meant to go below \"---\".\n\n> ---\n> list-objects.c | 31 +++++++++++++++++------------\n> revision.c | 1 -\n> revision.h | 11 ++++++++--\n> t/t6112-rev-list-filters-objects.sh | 10 ++++++++++\n> 4 files changed, 37 insertions(+), 16 deletions(-)\n>\n> diff --git a/list-objects.c b/list-objects.c\n> index 243192af5..7a1a0929d 100644\n> --- a/list-objects.c\n> +++ b/list-objects.c\n> @@ -53,7 +53,7 @@ static void process_blob(struct traversal_context *ctx,\n> \n> \tpathlen = path->len;\n> \tstrbuf_addstr(path, name);\n> -\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n> +\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n> \t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n> \t\t\t\t path->buf, &path->buf[pathlen],\n> \t\t\t\t ctx->filter_data);\n> @@ -120,17 +120,19 @@ static void process_tree_contents(struct traversal_context *ctx,\n> \t\t\t\tcontinue;\n> \t\t}\n> \n> -\t\tif (S_ISDIR(entry.mode))\n> -\t\t\tprocess_tree(ctx,\n> -\t\t\t\t lookup_tree(the_repository, entry.oid),\n> -\t\t\t\t base, entry.path);\n> +\t\tif (S_ISDIR(entry.mode)) {\n> +\t\t\tstruct tree *t = lookup_tree(the_repository, entry.oid);\n> +\t\t\tt->object.flags |= NOT_USER_GIVEN;\n> +\t\t\tprocess_tree(ctx, t, base, entry.path);\n> +\t\t}\n> \t\telse if (S_ISGITLINK(entry.mode))\n> \t\t\tprocess_gitlink(ctx, entry.oid->hash,\n> \t\t\t\t\tbase, entry.path);\n> -\t\telse\n> -\t\t\tprocess_blob(ctx,\n> -\t\t\t\t lookup_blob(the_repository, entry.oid),\n> -\t\t\t\t base, entry.path);\n> +\t\telse {\n> +\t\t\tstruct blob *b = lookup_blob(the_repository, entry.oid);\n> +\t\t\tb->object.flags |= NOT_USER_GIVEN;\n> +\t\t\tprocess_blob(ctx, b, base, entry.path);\n> +\t\t}\n> \t}\n> }\n> \n> @@ -171,7 +173,7 @@ static void process_tree(struct traversal_context *ctx,\n> \t}\n> \n> \tstrbuf_addstr(base, name);\n> -\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n> +\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n> \t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n> \t\t\t\t base->buf, &base->buf[baselen],\n> \t\t\t\t ctx->filter_data);\n> @@ -185,7 +187,7 @@ static void process_tree(struct traversal_context *ctx,\n> \tif (!failed_parse)\n> \t\tprocess_tree_contents(ctx, tree, base);\n> \n> -\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n> +\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n> \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n> \t\t\t\t base->buf, &base->buf[baselen],\n> \t\t\t\t ctx->filter_data);\n> @@ -301,8 +303,11 @@ static void do_traverse(struct traversal_context *ctx)\n> \t\t * an uninteresting boundary commit may not have its tree\n> \t\t * parsed yet, but we are not going to show them anyway\n> \t\t */\n> -\t\tif (get_commit_tree(commit))\n> -\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n> +\t\tif (get_commit_tree(commit)) {\n> +\t\t\tstruct tree *tree = get_commit_tree(commit);\n> +\t\t\ttree->object.flags |= NOT_USER_GIVEN;\n> +\t\t\tadd_pending_tree(ctx->revs, tree);\n> +\t\t}\n> \t\tctx->show_commit(commit, ctx->show_data);\n> \n> \t\tif (ctx->revs->tree_blobs_in_commit_order)\n> diff --git a/revision.c b/revision.c\n> index de4dce600..72d48a17f 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -175,7 +175,6 @@ static void add_pending_object_with_path(struct rev_info *revs,\n> \t\tstrbuf_release(&buf);\n> \t\treturn; /* do not add the commit itself */\n> \t}\n> -\tobj->flags |= USER_GIVEN;\n> \tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n> }\n> \n> diff --git a/revision.h b/revision.h\n> index 5910613cb..83e164039 100644\n> --- a/revision.h\n> +++ b/revision.h\n> @@ -21,9 +21,16 @@\n> #define SYMMETRIC_LEFT\t(1u<<8)\n> #define PATCHSAME\t(1u<<9)\n> #define BOTTOM\t\t(1u<<10)\n> -#define USER_GIVEN\t(1u<<25) /* given directly by the user */\n> +/*\n> + * Indicates object was reached by traversal. i.e. not given by user on\n> + * command-line or stdin.\n> + * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n> + * filtering trees and blobs, but it may be useful to support filtering commits\n> + * in the future.\n> + */\n> +#define NOT_USER_GIVEN\t(1u<<25)\n> #define TRACK_LINEAR\t(1u<<26)\n> -#define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n> +#define ALL_REV_FLAGS\t(((1u<<11)-1) | NOT_USER_GIVEN | TRACK_LINEAR)\n> \n> #define DECORATE_SHORT_REFS\t1\n> #define DECORATE_FULL_REFS\t2\n> diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\n> index c662c97db..2e07dadf0 100755\n> --- a/t/t6112-rev-list-filters-objects.sh\n> +++ b/t/t6112-rev-list-filters-objects.sh\n> @@ -30,6 +30,16 @@ test_expect_success 'verify blob:none omits all 5 blobs' '\n> \ttest_cmp observed expected\n> '\n> \n> +test_expect_success 'specify blob explicitly prevents filtering' '\n> +\tfile_3=$(git -C r1 ls-files -s file.3 \\\n> +\t\t| awk -f print_2.awk) &&\n> +\tfile_4=$(git -C r1 ls-files -s file.4 \\\n> +\t\t| awk -f print_2.awk) &&\n> +\tgit -C r1 rev-list HEAD --objects --filter=blob:none HEAD $file_3 >observed &&\n> +\tgrep -q \"$file_3\" observed &&\n> +\ttest_must_fail grep -q \"$file_4\" observed\n> +'\n> +\n> test_expect_success 'verify emitted+omitted == all' '\n> \tgit -C r1 rev-list HEAD --objects \\\n> \t\t| awk -f print_1.awk \\\n"},{"id":"358138","messageId":"xmqqr2hwasgv.fsf@gitster-ct.c.googlers.com","threadId":"49086","inReplyTo":"edba3791caf8bacc2f722f7874369f6776ecffe0.1536885967.git.matvore@google.com","subject":"Re: [PATCH v8 7/7] list-objects-filter: implement filter tree:0","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-14T17:39:44Z","receivedAt":"2018-09-14T17:39:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthew DeVore writes:\n\n> diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\n> index c0e2bd6a0..14f251de4 100644\n> --- a/list-objects-filter-options.c\n> +++ b/list-objects-filter-options.c\n> @@ -50,6 +50,20 @@ static int gently_parse_list_objects_filter(\n> \t\t\treturn 0;\n> \t\t}\n> \n> +\t} else if (skip_prefix(arg, \"tree:\", &v0)) {\n> +\t\tunsigned long depth;\n> +\t\tif (!git_parse_ulong(v0, &depth) || depth != 0) {\n> +\t\t\tif (errbuf) {\n> +\t\t\t\tstrbuf_init(errbuf, 0);\n> +\t\t\t\tstrbuf_addstr(\n> +\t\t\t\t\terrbuf,\n> +\t\t\t\t\t_(\"only 'tree:0' is supported\"));\n\nThis is not a new issue with this patch, but I think strbuf_init()\nat the location of filling done like this is a bad idea. If the\ncaller gave you an errbuf that is pre-filled with something, we'd\nleak memory and lose information. It only makes sense to _init() if\nthe caller gave us an uninitialized garbage (or a strbuf that has\njust been initialized and is empty). \n\nThe existing callers seem to do STRBUF_INIT before passing it to\nthis function, so we probably should not do strbuf_init() here (and\nother two places in this function) and simply add to it.\n\n> diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\n> index bbbe7537d..8eeb85fbc 100755\n> --- a/t/t5616-partial-clone.sh\n> +++ b/t/t5616-partial-clone.sh\n> @@ -154,6 +154,44 @@ test_expect_success 'partial clone with transfer.fsckobjects=1 uses index-pack -\n> \tgrep \"git index-pack.*--fsck-objects\" trace\n> '\n> \n> +test_expect_success 'use fsck before and after manually fetching a missing subtree' '\n> +\t# push new commit so server has a subtree\n> +\tmkdir src/dir &&\n> +\techo \"in dir\" >src/dir/file.txt &&\n> +\tgit -C src add dir/file.txt &&\n> +\tgit -C src commit -m \"file in dir\" &&\n> +\tgit -C src push -u srv master &&\n> +\tSUBTREE=$(git -C src rev-parse HEAD:dir) &&\n> +\n> +\trm -rf dst &&\n> +\tgit clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n> +\tgit -C dst fsck &&\n> +\n> +\t# Make sure we only have commits, and all trees and blobs are missing.\n> +\tgit -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n> +\tawk -f print_1.awk fetched_objects \\\n> +\t\t| xargs -n1 git -C dst cat-file -t >fetched_types &&\n\nBreak line after pipe \"|\", not before, and lose the backslash. You\ndo not need to over-indent the command on the downstream of the\npipe, i.e.\n\n\tawk ... |\n\txargs -n1 git -C ... &&\n\nSame comment applies elsewhere in this patch, not limited to this file.\n\n> +\tsort fetched_types -u >unique_types.observed &&\n\nMake it a habit not to add dashed options after real arguments, i.e.\n\n\tsort -u fetched_types\n\n> +\techo commit >unique_types.expected &&\n> +\ttest_cmp unique_types.observed unique_types.expected &&\n\nAlways compare \"expect\" with \"actual\", not in the reverse order, i.e.\n\n\ttest_cmp expect actual\n\nnot\n\n\ttest_cmp actual expect\n\nThis is important because test_cmp reports failures by showing you\nan output of \"diff expect actual\" and from \"sh t5616-part*.sh -v\"\nyou can see what additional/excess things were produced by the test\nover what is expected, prefixed with \"+\", and what your code failed\nto produce are shown prefixed with \"-\".\n\nThanks.\n\n"},{"id":"358139","messageId":"xmqqmuskas3a.fsf@gitster-ct.c.googlers.com","threadId":"49086","inReplyTo":"xmqqr2hwasgv.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v8 7/7] list-objects-filter: implement filter tree:0","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-14T17:47:53Z","receivedAt":"2018-09-14T17:47:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\n>> index bbbe7537d..8eeb85fbc 100755\n>> --- a/t/t5616-partial-clone.sh\n>> +++ b/t/t5616-partial-clone.sh\n>> ...\n>\n> Break line after pipe \"|\", not before, and lose the backslash. You\n> do not need to over-indent the command on the downstream of the\n> pipe, i.e.\n>\n> \tawk ... |\n> \txargs -n1 git -C ... &&\n>\n> Same comment applies elsewhere in this patch, not limited to this file.\n>\n>> +\tsort fetched_types -u >unique_types.observed &&\n>\n> Make it a habit not to add dashed options after real arguments, i.e.\n>\n> \tsort -u fetched_types\n>\n>> +\techo commit >unique_types.expected &&\n>> +\ttest_cmp unique_types.observed unique_types.expected &&\n>\n> Always compare \"expect\" with \"actual\", not in the reverse order, i.e.\n>\n> \ttest_cmp expect actual\n>\n> not\n>\n> \ttest_cmp actual expect\n>\n> This is important because test_cmp reports failures by showing you\n> an output of \"diff expect actual\" and from \"sh t5616-part*.sh -v\"\n> you can see what additional/excess things were produced by the test\n> over what is expected, prefixed with \"+\", and what your code failed\n> to produce are shown prefixed with \"-\".\n\nI notice that patches to other files like 6112 in this series also\nspread the above mistakes from existing lines. Please do not view\nwhat you see in these two test scripts before you start touching as\na good example to follow---rather, treat them as antipattern X-<.\n5616 is not as bad as 6112, but they both need to be cleaned up.\n\nWe could alternatively do a post clean-up, but ideally we should\nfirst have a clean-up patch before this series to co.\n\nThanks.\n"},{"id":"358152","messageId":"CAMfpvhLkn1wbDwzQycW4SX92qmyof6n3gMXe-f-5JdQKk4P28g@mail.gmail.com","threadId":"49086","inReplyTo":"xmqqzhwkat8i.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v8 5/7] revision: mark non-user-given objects instead","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-14T20:08:36Z","receivedAt":"2018-09-14T20:08:51Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Fri, Sep 14, 2018 at 10:23 AM Junio C Hamano wrote:\n>\n> Matthew DeVore writes:\n>\n> > Signed-off-by: Matthew DeVore \n> >\n> > fixup of 6defd70de\n>\n> That's probably meant to go below \"---\".\n>\n\nThat line shouldn't be there at all, sorry!\n\nIt came from me putting that text in a commit which was meant to be a\nfixup of another commit when I ran rebase -i. I asked rebase to make\nit a \"squash\" so I could edit the commit message of the earlier commit\n(6defd70de). Then rebase merged the two descriptions and let me edit\nthem, but I didn't remember to delete the latter commit's message.\n\nI probably should have made the earlier commit (6defd70de) a \"reword\",\nand the later commit a \"fixup\", rather than \"pick\" followed by\n\"squash\"\n"},{"id":"358176","messageId":"CAMfpvhLLd+rC+vD4Sj-ojRGS6_GF84PAEQt8TCVCfp4JHST+oQ@mail.gmail.com","threadId":"49086","inReplyTo":"xmqqmuskas3a.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v8 7/7] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-15T00:41:04Z","receivedAt":"2018-09-15T00:41:20Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Fri, Sep 14, 2018 at 10:47 AM Junio C Hamano wrote:\n>\n> Junio C Hamano writes:\n>\n> > Break line after pipe \"|\", not before, and lose the backslash. You\n> > do not need to over-indent the command on the downstream of the\n> > pipe, i.e.\n> >\n> > awk ... |\n> > xargs -n1 git -C ... &&\n> >\n> > Same comment applies elsewhere in this patch, not limited to this file.\n> >\n> >> + sort fetched_types -u >unique_types.observed &&\n> >\n> > Make it a habit not to add dashed options after real arguments, i.e.\n> >\n> > sort -u fetched_types\n> >\nDone. I'm not sure why I made this mistake, since I usually prefer to\norder flags before positional args. I didn't actually clean this up in\nexisting code as I did other mistakes, since it is very hard to find\nand do thoroughly.\n\n> >> + echo commit >unique_types.expected &&\n> >> + test_cmp unique_types.observed unique_types.expected &&\n> >\n> > Always compare \"expect\" with \"actual\", not in the reverse order, i.e.\n> >\n> > test_cmp expect actual\n> >\n> > not\n> >\n> > test_cmp actual expect\n> >\nDone.\n\n> > This is important because test_cmp reports failures by showing you\n> > an output of \"diff expect actual\" and from \"sh t5616-part*.sh -v\"\n> > you can see what additional/excess things were produced by the test\n> > over what is expected, prefixed with \"+\", and what your code failed\n> > to produce are shown prefixed with \"-\".\nHmm... I didn't know aout the -v flag. That's quite good to know, thanks!\n\n>\n> I notice that patches to other files like 6112 in this series also\n> spread the above mistakes from existing lines. Please do not view\n> what you see in these two test scripts before you start touching as\n> a good example to follow---rather, treat them as antipattern X-<.\n> 5616 is not as bad as 6112, but they both need to be cleaned up.\n>\n> We could alternatively do a post clean-up, but ideally we should\n> first have a clean-up patch before this series to co.\n\nI cleaned up existing tests in a new patchset here:\nhttps://public-inbox.org/git/cover.1536969438.git.matvore@google.com/T/#t\n- that new patch corrects the pipe placement and test_cmp argument\nordering.\n\nThere is no dependency between this patchset and the new one, though I\nassume you want to commit the clean-up once first so maintain\nconsistency.\n\nHere is an interdiff for this particular patch series (I replaced \\t\nwith 8 spaces so it would be readable after my mail client mangles\nit):\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 14f251de4..e8da2e858 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -30,7 +30,6 @@ static int gently_parse_list_objects_filter(\n\n if (filter_options->choice) {\n if (errbuf) {\n- strbuf_init(errbuf, 0);\n strbuf_addstr(\n errbuf,\n _(\"multiple filter-specs cannot be combined\"));\n@@ -54,7 +53,6 @@ static int gently_parse_list_objects_filter(\n unsigned long depth;\n if (!git_parse_ulong(v0, &depth) || depth != 0) {\n if (errbuf) {\n- strbuf_init(errbuf, 0);\n strbuf_addstr(\n errbuf,\n _(\"only 'tree:0' is supported\"));\n@@ -85,10 +83,9 @@ static int gently_parse_list_objects_filter(\n return 0;\n }\n\n- if (errbuf) {\n- strbuf_init(errbuf, 0);\n+ if (errbuf)\n strbuf_addf(errbuf, \"invalid filter-spec '%s'\", arg);\n- }\n+\n memset(filter_options, 0, sizeof(*filter_options));\n return 1;\n }\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex f02b9ae37..5bc5b4445 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -216,7 +216,7 @@ test_expect_success 'missing non-root tree object\nand rev-list' '\n rm -rf repo &&\n test_create_repo repo &&\n mkdir repo/dir &&\n- echo foo > repo/dir/foo &&\n+ echo foo >repo/dir/foo &&\n git -C repo add dir/foo &&\n git -C repo commit -m \"commit dir/foo\" &&\n\ndiff --git a/t/t5317-pack-objects-filter-objects.sh\nb/t/t5317-pack-objects-filter-objects.sh\nindex 7a4d49ea1..510d3537f 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -61,7 +61,7 @@ test_expect_success 'verify normal and blob:none\npackfiles have same commits/tre\n\n test_expect_success 'get an error for missing tree object' '\n git init r5 &&\n- echo foo > r5/foo &&\n+ echo foo >r5/foo &&\n git -C r5 add foo &&\n git -C r5 commit -m \"foo\" &&\n del=$(git -C r5 rev-parse HEAD^{tree} | sed \"s|..|&/|\") &&\n@@ -97,7 +97,7 @@ test_expect_success 'grab tree directly when using tree:0' '\n git -C r1 verify-pack -v ../commitsonly.pack >objs &&\n awk \"/tree|blob/{print \\$1}\" objs >trees_and_blobs &&\n git -C r1 rev-parse HEAD: >expected &&\n- test_cmp trees_and_blobs expected\n+ test_cmp expected trees_and_blobs\n '\n\n # Test blob:limit=[kmg] filter.\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 8eeb85fbc..7b6294ca5 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -169,11 +169,12 @@ test_expect_success 'use fsck before and after\nmanually fetching a missing subtr\n\n # Make sure we only have commits, and all trees and blobs are missing.\n git -C dst rev-list master --missing=allow-any --objects\n>fetched_objects &&\n- awk -f print_1.awk fetched_objects \\\n- | xargs -n1 git -C dst cat-file -t >fetched_types &&\n- sort fetched_types -u >unique_types.observed &&\n+ awk -f print_1.awk fetched_objects |\n+ xargs -n1 git -C dst cat-file -t >fetched_types &&\n+\n+ sort -u fetched_types >unique_types.observed &&\n echo commit >unique_types.expected &&\n- test_cmp unique_types.observed unique_types.expected &&\n+ test_cmp unique_types.expected unique_types.observed &&\n\n # Auto-fetch a tree with cat-file.\n git -C dst cat-file -p $SUBTREE >tree_contents &&\n@@ -185,11 +186,13 @@ test_expect_success 'use fsck before and after\nmanually fetching a missing subtr\n # Auto-fetch all remaining trees and blobs with --missing=error\n git -C dst rev-list master --missing=error --objects\n>fetched_objects &&\n test_line_count = 70 fetched_objects &&\n- awk -f print_1.awk fetched_objects \\\n- | xargs -n1 git -C dst cat-file -t >fetched_types &&\n- sort fetched_types -u >unique_types.observed &&\n+\n+ awk -f print_1.awk fetched_objects |\n+ xargs -n1 git -C dst cat-file -t >fetched_types &&\n+\n+ sort -u fetched_types >unique_types.observed &&\n printf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n- test_cmp unique_types.observed unique_types.expected\n+ test_cmp unique_types.expected unique_types.observed\n '\n\n test_expect_success 'partial clone fetches blobs pointed to by refs\neven if normally filtered out' '\ndiff --git a/t/t6112-rev-list-filters-objects.sh\nb/t/t6112-rev-list-filters-objects.sh\nindex a989a7082..6e5c41a68 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -31,11 +31,13 @@ test_expect_success 'verify blob:none omits all 5 blobs' '\n '\n\n test_expect_success 'specify blob explicitly prevents filtering' '\n- file_3=$(git -C r1 ls-files -s file.3 \\\n- | awk -f print_2.awk) &&\n- file_4=$(git -C r1 ls-files -s file.4 \\\n- | awk -f print_2.awk) &&\n- git -C r1 rev-list HEAD --objects --filter=blob:none HEAD\n$file_3 >observed &&\n+ file_3=$(git -C r1 ls-files -s file.3 |\n+ awk -f print_2.awk) &&\n+\n+ file_4=$(git -C r1 ls-files -s file.4 |\n+ awk -f print_2.awk) &&\n+\n+ git -C r1 rev-list --objects --filter=blob:none HEAD $file_3\n>observed &&\n grep -q \"$file_3\" observed &&\n test_must_fail grep -q \"$file_4\" observed\n '\n@@ -225,13 +227,14 @@ test_expect_success 'rev-list W/ --missing=print\nand --missing=allow-any for tre\n # Test tree:0 filter.\n\n test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n- git -C r3 rev-list HEAD --quiet --objects\n--filter-print-omitted --filter=tree:0 \\\n- | awk -f print_1.awk \\\n- | sed s/~// \\\n- | xargs -n1 git -C r3 cat-file -t \\\n- | sort -u >filtered_types &&\n- printf \"blob\\ntree\\n\" > expected &&\n- test_cmp filtered_types expected\n+ git -C r3 rev-list HEAD --quiet --objects\n--filter-print-omitted --filter=tree:0 |\n+ awk -f print_1.awk |\n+ sed s/~// |\n+ xargs -n1 git -C r3 cat-file -t |\n+ sort -u >filtered_types &&\n+\n+ printf \"blob\\ntree\\n\" >expected &&\n+ test_cmp expected filtered_types\n '\n\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n"},{"id":"358646","messageId":"cover.1537561420.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"[PATCH v9 0/8] filter: support for excluding all trees and blobs","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-21T20:31:56Z","receivedAt":"2018-09-21T20:32:25Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Since v8, I cleaned up the test scripts in the following ways:\n - correct order of expect/actual arguments to test_cmp\n - correct pipe placement\n - put flags before positional arguments\n\nAlso, removed some junk in the commit message of the 5th patch.\n\nThank you,\n\nMatthew DeVore (8):\n list-objects: store common func args in struct\n list-objects: refactor to process_tree_contents\n list-objects: always parse trees gently\n rev-list: handle missing tree objects properly\n revision: mark non-user-given objects instead\n list-objects-filter: use BUG rather than die\n list-objects-filter-options: do not over-strbuf_init\n list-objects-filter: implement filter tree:0\n\n Documentation/rev-list-options.txt | 5 +\n builtin/rev-list.c | 11 +-\n list-objects-filter-options.c | 19 +-\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 60 ++++++-\n list-objects.c | 232 +++++++++++++------------\n revision.c | 1 -\n revision.h | 26 ++-\n t/t0410-partial-clone.sh | 45 +++++\n t/t5317-pack-objects-filter-objects.sh | 41 +++++\n t/t5616-partial-clone.sh | 41 +++++\n t/t6112-rev-list-filters-objects.sh | 42 +++++\n 12 files changed, 396 insertions(+), 128 deletions(-)\n\n-- \n2.19.0.444.g18242da7ef-goog\n\n"},{"id":"358647","messageId":"3bff094363c7887bfe6547e520cbe83097cf6f73.1537561420.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1537561420.git.matvore@google.com","subject":"[PATCH v9 1/8] list-objects: store common func args in struct","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-21T20:31:57Z","receivedAt":"2018-09-21T20:32:29Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will make utility functions easier to create, as done by the next\npatch.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 158 +++++++++++++++++++++++--------------------------\n 1 file changed, 74 insertions(+), 84 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex c99c47ac1..584518a3f 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -12,20 +12,25 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n \n-static void process_blob(struct rev_info *revs,\n+struct traversal_context {\n+\tstruct rev_info *revs;\n+\tshow_object_fn show_object;\n+\tshow_commit_fn show_commit;\n+\tvoid *show_data;\n+\tfilter_object_fn filter_fn;\n+\tvoid *filter_data;\n+};\n+\n+static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \n-\tif (!revs->blob_objects)\n+\tif (!ctx->revs->blob_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad blob object\");\n@@ -41,21 +46,21 @@ static void process_blob(struct rev_info *revs,\n \t * may cause the actual filter to report an incomplete list\n \t * of missing objects.\n \t */\n-\tif (revs->exclude_promisor_objects &&\n+\tif (ctx->revs->exclude_promisor_objects &&\n \t !has_object_file(&obj->oid) &&\n \t is_promisor_object(&obj->oid))\n \t\treturn;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BLOB, obj,\n-\t\t\t path->buf, &path->buf[pathlen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n+\t\t\t\t path->buf, &path->buf[pathlen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, path->buf, cb_data);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -81,26 +86,21 @@ static void process_blob(struct rev_info *revs,\n * the link, and how to do it. Whether it necessarily makes\n * any sense what-so-ever to ever do that is another issue.\n */\n-static void process_gitlink(struct rev_info *revs,\n+static void process_gitlink(struct traversal_context *ctx,\n \t\t\t const unsigned char *sha1,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data)\n+\t\t\t const char *name)\n {\n \t/* Nothing to do */\n }\n \n-static void process_tree(struct rev_info *revs,\n+static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &tree->object;\n+\tstruct rev_info *revs = ctx->revs;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n@@ -133,14 +133,14 @@ static void process_tree(struct rev_info *revs,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BEGIN_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, base->buf, cb_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -157,29 +157,25 @@ static void process_tree(struct rev_info *revs,\n \t\t}\n \n \t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(revs,\n+\t\t\tprocess_tree(ctx,\n \t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(revs, entry.oid->hash,\n-\t\t\t\t\tshow, base, entry.path,\n-\t\t\t\t\tcb_data);\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n \t\telse\n-\t\t\tprocess_blob(revs,\n+\t\t\tprocess_blob(ctx,\n \t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t}\n \n-\tif (!(obj->flags & USER_GIVEN) && filter_fn) {\n-\t\tr = filter_fn(LOFS_END_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \t\tif (r & LOFR_MARK_SEEN)\n \t\t\tobj->flags |= SEEN;\n \t\tif (r & LOFR_DO_SHOW)\n-\t\t\tshow(obj, base->buf, cb_data);\n+\t\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \t}\n \n \tstrbuf_setlen(base, baselen);\n@@ -242,19 +238,15 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n-static void traverse_trees_and_blobs(struct rev_info *revs,\n-\t\t\t\t struct strbuf *base,\n-\t\t\t\t show_object_fn show_object,\n-\t\t\t\t void *show_data,\n-\t\t\t\t filter_object_fn filter_fn,\n-\t\t\t\t void *filter_data)\n+static void traverse_trees_and_blobs(struct traversal_context *ctx,\n+\t\t\t\t struct strbuf *base)\n {\n \tint i;\n \n \tassert(base->len == 0);\n \n-\tfor (i = 0; i < revs->pending.nr; i++) {\n-\t\tstruct object_array_entry *pending = revs->pending.objects + i;\n+\tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n+\t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n@@ -262,62 +254,49 @@ static void traverse_trees_and_blobs(struct rev_info *revs,\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tshow_object(obj, name, show_data);\n+\t\t\tctx->show_object(obj, name, ctx->show_data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(revs, (struct tree *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(revs, (struct blob *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n \t\t oid_to_hex(&obj->oid), name);\n \t}\n-\tobject_array_clear(&revs->pending);\n+\tobject_array_clear(&ctx->revs->pending);\n }\n \n-static void do_traverse(struct rev_info *revs,\n-\t\t\tshow_commit_fn show_commit,\n-\t\t\tshow_object_fn show_object,\n-\t\t\tvoid *show_data,\n-\t\t\tfilter_object_fn filter_fn,\n-\t\t\tvoid *filter_data)\n+static void do_traverse(struct traversal_context *ctx)\n {\n \tstruct commit *commit;\n \tstruct strbuf csp; /* callee's scratch pad */\n \tstrbuf_init(&csp, PATH_MAX);\n \n-\twhile ((commit = get_revision(revs)) != NULL) {\n+\twhile ((commit = get_revision(ctx->revs)) != NULL) {\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n \t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(revs, get_commit_tree(commit));\n-\t\tshow_commit(commit, show_data);\n+\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tctx->show_commit(commit, ctx->show_data);\n \n-\t\tif (revs->tree_blobs_in_commit_order)\n+\t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n \t\t\t * NEEDSWORK: Adding the tree and then flushing it here\n \t\t\t * needs a reallocation for each commit. Can we pass the\n \t\t\t * tree directory without allocation churn?\n \t\t\t */\n-\t\t\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t\t\t show_object, show_data,\n-\t\t\t\t\t\t filter_fn, filter_data);\n+\t\t\ttraverse_trees_and_blobs(ctx, &csp);\n \t}\n-\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t show_object, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\ttraverse_trees_and_blobs(ctx, &csp);\n \tstrbuf_release(&csp);\n }\n \n@@ -326,7 +305,14 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t\t show_object_fn show_object,\n \t\t\t void *show_data)\n {\n-\tdo_traverse(revs, show_commit, show_object, show_data, NULL, NULL);\n+\tstruct traversal_context ctx;\n+\tctx.revs = revs;\n+\tctx.show_commit = show_commit;\n+\tctx.show_object = show_object;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\tctx.filter_data = NULL;\n+\tdo_traverse(&ctx);\n }\n \n void traverse_commit_list_filtered(\n@@ -337,14 +323,18 @@ void traverse_commit_list_filtered(\n \tvoid *show_data,\n \tstruct oidset *omitted)\n {\n-\tfilter_object_fn filter_fn = NULL;\n+\tstruct traversal_context ctx;\n \tfilter_free_fn filter_free_fn = NULL;\n-\tvoid *filter_data = NULL;\n-\n-\tfilter_data = list_objects_filter__init(omitted, filter_options,\n-\t\t\t\t\t\t&filter_fn, &filter_free_fn);\n-\tdo_traverse(revs, show_commit, show_object, show_data,\n-\t\t filter_fn, filter_data);\n-\tif (filter_data && filter_free_fn)\n-\t\tfilter_free_fn(filter_data);\n+\n+\tctx.revs = revs;\n+\tctx.show_object = show_object;\n+\tctx.show_commit = show_commit;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\n+\tctx.filter_data = list_objects_filter__init(omitted, filter_options,\n+\t\t\t\t\t\t &ctx.filter_fn, &filter_free_fn);\n+\tdo_traverse(&ctx);\n+\tif (ctx.filter_data && filter_free_fn)\n+\t\tfilter_free_fn(ctx.filter_data);\n }\n-- \n2.19.0.444.g18242da7ef-goog\n\n"},{"id":"358648","messageId":"42ad5623bab52b8207831eab4a4f27e47a318e36.1537561420.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1537561420.git.matvore@google.com","subject":"[PATCH v9 2/8] list-objects: refactor to process_tree_contents","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-21T20:31:58Z","receivedAt":"2018-09-21T20:32:29Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will be used in a follow-up patch to reduce indentation needed when\ninvoking the logic conditionally. i.e. rather than:\n\nif (foo) {\n\twhile (...) {\n\t\t/* this is very indented */\n\t}\n}\n\nwe will have:\n\nif (foo)\n\tprocess_tree_contents(...);\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 68 ++++++++++++++++++++++++++++++--------------------\n 1 file changed, 41 insertions(+), 27 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 584518a3f..ccc529e5e 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -94,6 +94,46 @@ static void process_gitlink(struct traversal_context *ctx,\n \t/* Nothing to do */\n }\n \n+static void process_tree(struct traversal_context *ctx,\n+\t\t\t struct tree *tree,\n+\t\t\t struct strbuf *base,\n+\t\t\t const char *name);\n+\n+static void process_tree_contents(struct traversal_context *ctx,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *base)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tenum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ?\n+\t\tall_entries_interesting : entry_not_interesting;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tif (match != all_entries_interesting) {\n+\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n+\t\t\t\t\t\t &ctx->revs->diffopt.pathspec);\n+\t\t\tif (match == all_entries_not_interesting)\n+\t\t\t\tbreak;\n+\t\t\tif (match == entry_not_interesting)\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (S_ISDIR(entry.mode))\n+\t\t\tprocess_tree(ctx,\n+\t\t\t\t lookup_tree(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t\telse if (S_ISGITLINK(entry.mode))\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash,\n+\t\t\t\t\tbase, entry.path);\n+\t\telse\n+\t\t\tprocess_blob(ctx,\n+\t\t\t\t lookup_blob(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t}\n+}\n+\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n@@ -101,10 +141,6 @@ static void process_tree(struct traversal_context *ctx,\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n-\tstruct tree_desc desc;\n-\tstruct name_entry entry;\n-\tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n-\t\tall_entries_interesting: entry_not_interesting;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \tint gently = revs->ignore_missing_links ||\n@@ -144,29 +180,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tinit_tree_desc(&desc, tree->buffer, tree->size);\n-\n-\twhile (tree_entry(&desc, &entry)) {\n-\t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n-\t\t\t\t\t\t &revs->diffopt.pathspec);\n-\t\t\tif (match == all_entries_not_interesting)\n-\t\t\t\tbreak;\n-\t\t\tif (match == entry_not_interesting)\n-\t\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t}\n+\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n-- \n2.19.0.444.g18242da7ef-goog\n\n"},{"id":"358649","messageId":"0d1285a0ec412846e5417ec524a866f6b2c39d82.1537561420.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1537561420.git.matvore@google.com","subject":"[PATCH v9 3/8] list-objects: always parse trees gently","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-21T20:31:59Z","receivedAt":"2018-09-21T20:32:31Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"If parsing fails when revs->ignore_missing_links and\nrevs->exclude_promisor_objects are both false, we print the OID anyway\nin the die(\"bad tree object...\") call, so any message printed by\nparse_tree_gently() is superfluous.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex ccc529e5e..f9b51db7a 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,8 +143,6 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n-\tint gently = revs->ignore_missing_links ||\n-\t\t revs->exclude_promisor_objects;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -152,7 +150,7 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, gently) < 0) {\n+\tif (parse_tree_gently(tree, 1) < 0) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n-- \n2.19.0.444.g18242da7ef-goog\n\n"},{"id":"358650","messageId":"a712bad4860f578a339f3d923b1476c11eb4d291.1537561420.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1537561420.git.matvore@google.com","subject":"[PATCH v9 4/8] rev-list: handle missing tree objects properly","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-21T20:32:00Z","receivedAt":"2018-09-21T20:32:34Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Previously, we assumed only blob objects could be missing. This patch\nmakes rev-list handle missing trees like missing blobs. The --missing=*\nand --exclude-promisor-objects flags now work for trees as they already\ndo for blobs. This is demonstrated in t6112.\n\nSigned-off-by: Matthew DeVore \n---\n builtin/rev-list.c | 11 ++++---\n list-objects.c | 11 +++++--\n revision.h | 15 +++++++++\n t/t0410-partial-clone.sh | 45 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 13 ++++++++\n t/t6112-rev-list-filters-objects.sh | 17 ++++++++++\n 6 files changed, 105 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 5b07f3f4a..49d6deed7 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -6,6 +6,7 @@\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"object.h\"\n #include \"object-store.h\"\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n@@ -209,7 +210,8 @@ static inline void finish_object__ma(struct object *obj)\n \t */\n \tswitch (arg_missing_action) {\n \tcase MA_ERROR:\n-\t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n+\t\tdie(\"missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tcase MA_ALLOW_ANY:\n@@ -222,8 +224,8 @@ static inline void finish_object__ma(struct object *obj)\n \tcase MA_ALLOW_PROMISOR:\n \t\tif (is_promisor_object(&obj->oid))\n \t\t\treturn;\n-\t\tdie(\"unexpected missing blob object '%s'\",\n-\t\t oid_to_hex(&obj->oid));\n+\t\tdie(\"unexpected missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tdefault:\n@@ -235,7 +237,7 @@ static inline void finish_object__ma(struct object *obj)\n static int finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\tif (!has_object_file(&obj->oid)) {\n \t\tfinish_object__ma(obj);\n \t\treturn 1;\n \t}\n@@ -373,6 +375,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tinit_revisions(&revs, prefix);\n \trevs.abbrev = DEFAULT_ABBREV;\n \trevs.commit_format = CMIT_FMT_UNSPECIFIED;\n+\trevs.do_not_die_on_missing_tree = 1;\n \n \t/*\n \t * Scan the argument list before invoking setup_revisions(), so that we\ndiff --git a/list-objects.c b/list-objects.c\nindex f9b51db7a..243192af5 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,6 +143,7 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tint failed_parse;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -150,7 +151,9 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, 1) < 0) {\n+\n+\tfailed_parse = parse_tree_gently(tree, 1);\n+\tif (failed_parse) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n@@ -163,7 +166,8 @@ static void process_tree(struct traversal_context *ctx,\n \t\t is_promisor_object(&obj->oid))\n \t\t\treturn;\n \n-\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n+\t\tif (!revs->do_not_die_on_missing_tree)\n+\t\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \n \tstrbuf_addstr(base, name);\n@@ -178,7 +182,8 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tprocess_tree_contents(ctx, tree, base);\n+\tif (!failed_parse)\n+\t\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\ndiff --git a/revision.h b/revision.h\nindex 007278cc1..5910613cb 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -126,6 +126,21 @@ struct rev_info {\n \t\t\tline_level_traverse:1,\n \t\t\ttree_blobs_in_commit_order:1,\n \n+\t\t\t/*\n+\t\t\t * Blobs are shown without regard for their existence.\n+\t\t\t * But not so for trees: unless exclude_promisor_objects\n+\t\t\t * is set and the tree in question is a promisor object;\n+\t\t\t * OR ignore_missing_links is set, the revision walker\n+\t\t\t * dies with a \"bad tree object HASH\" message when\n+\t\t\t * encountering a missing tree. For callers that can\n+\t\t\t * handle missing trees and want them to be filterable\n+\t\t\t * and showable, set this to true. The revision walker\n+\t\t\t * will filter and show such a missing tree as usual,\n+\t\t\t * but will not attempt to recurse into this tree\n+\t\t\t * object.\n+\t\t\t */\n+\t\t\tdo_not_die_on_missing_tree:1,\n+\n \t\t\t/* for internal use only */\n \t\t\texclude_promisor_objects:1;\n \ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 128130066..5bc5b4445 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -186,6 +186,51 @@ test_expect_success 'rev-list stops traversal at missing and promised commit' '\n \t! grep $FOO out\n '\n \n+test_expect_success 'missing tree objects with --missing=allow-promisor and --exclude-promisor-objects' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo foo &&\n+\ttest_commit -C repo bar &&\n+\ttest_commit -C repo baz &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse bar^{tree}) &&\n+\tpromise_and_delete $(git -C repo rev-parse foo^{tree}) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-promisor --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 3 commits, 3 blobs, and 1 tree\n+\ttest_line_count = 7 objs &&\n+\n+\t# Do the same for --exclude-promisor-objects, but with all trees gone.\n+\tpromise_and_delete $(git -C repo rev-parse baz^{tree}) &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 3 commits, no blobs or trees\n+\ttest_line_count = 3 objs\n+'\n+\n+test_expect_success 'missing non-root tree object and rev-list' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\tmkdir repo/dir &&\n+\techo foo >repo/dir/foo &&\n+\tgit -C repo add dir/foo &&\n+\tgit -C repo commit -m \"commit dir/foo\" &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse HEAD:dir) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 1 commit and 1 tree\n+\ttest_line_count = 2 objs\n+'\n+\n test_expect_success 'rev-list stops traversal at missing and promised tree' '\n \trm -rf repo &&\n \ttest_create_repo repo &&\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 6710c8bc8..9839b48c1 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -59,6 +59,19 @@ test_expect_success 'verify normal and blob:none packfiles have same commits/tre\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'get an error for missing tree object' '\n+\tgit init r5 &&\n+\techo foo >r5/foo &&\n+\tgit -C r5 add foo &&\n+\tgit -C r5 commit -m \"foo\" &&\n+\tdel=$(git -C r5 rev-parse HEAD^{tree} | sed \"s|..|&/|\") &&\n+\trm r5/.git/objects/$del &&\n+\ttest_must_fail git -C r5 pack-objects --rev --stdout 2>bad_tree <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgrep -q \"bad tree object\" bad_tree\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex d4ff0b3be..c662c97db 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -195,6 +195,23 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for trees' '\n+\tTREE=$(git -C r3 rev-parse HEAD:dir1) &&\n+\n+\trm r3/.git/objects/$(echo $TREE | sed \"s|^..|&/|\") &&\n+\n+\tgit -C r3 rev-list --quiet --missing=print --objects HEAD >missing_objs 2>rev_list_err &&\n+\techo \"?$TREE\" >expected &&\n+\ttest_cmp expected missing_objs &&\n+\n+\t# do not complain when a missing tree cannot be parsed\n+\ttest_must_be_empty rev_list_err &&\n+\n+\tgit -C r3 rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n+\t! grep $TREE objs &&\n+\ttest_must_be_empty rev_list_err\n+'\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.19.0.444.g18242da7ef-goog\n\n"},{"id":"358651","messageId":"6fb22953a334327a48881f691481981af925d56a.1537561420.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1537561420.git.matvore@google.com","subject":"[PATCH v9 5/8] revision: mark non-user-given objects instead","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-21T20:32:01Z","receivedAt":"2018-09-21T20:32:36Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Currently, list-objects.c incorrectly treats all root trees of commits\nas USER_GIVEN. Also, it would be easier to mark objects that are\nnon-user-given instead of user-given, since the places in the code\nwhere we access an object through a reference are more obvious than\nthe places where we access an object that was given by the user.\n\nResolve these two problems by introducing a flag NOT_USER_GIVEN that\nmarks blobs and trees that are non-user-given, replacing USER_GIVEN.\n(Only blobs and trees are marked because this mark is only used when\nfiltering objects, and filtering of other types of objects is not\nsupported yet.)\n\nThis fixes a bug in that git rev-list behaved differently from git\npack-objects. pack-objects would *not* filter objects given explicitly\non the command line and rev-list would filter. This was because the two\ncommands used a different function to add objects to the rev_info\nstruct. This seems to have been an oversight, and pack-objects has the\ncorrect behavior, so I added a test to make sure that rev-list now\nbehaves properly.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 31 +++++++++++++++++------------\n revision.c | 1 -\n revision.h | 11 ++++++++--\n t/t6112-rev-list-filters-objects.sh | 12 +++++++++++\n 4 files changed, 39 insertions(+), 16 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 243192af5..7a1a0929d 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -53,7 +53,7 @@ static void process_blob(struct traversal_context *ctx,\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n \t\t\t\t path->buf, &path->buf[pathlen],\n \t\t\t\t ctx->filter_data);\n@@ -120,17 +120,19 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\tcontinue;\n \t\t}\n \n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\tstruct tree *t = lookup_tree(the_repository, entry.oid);\n+\t\t\tt->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_tree(ctx, t, base, entry.path);\n+\t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid->hash,\n \t\t\t\t\tbase, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\telse {\n+\t\t\tstruct blob *b = lookup_blob(the_repository, entry.oid);\n+\t\t\tb->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_blob(ctx, b, base, entry.path);\n+\t\t}\n \t}\n }\n \n@@ -171,7 +173,7 @@ static void process_tree(struct traversal_context *ctx,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -185,7 +187,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (!failed_parse)\n \t\tprocess_tree_contents(ctx, tree, base);\n \n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -301,8 +303,11 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n-\t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tif (get_commit_tree(commit)) {\n+\t\t\tstruct tree *tree = get_commit_tree(commit);\n+\t\t\ttree->object.flags |= NOT_USER_GIVEN;\n+\t\t\tadd_pending_tree(ctx->revs, tree);\n+\t\t}\n \t\tctx->show_commit(commit, ctx->show_data);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\ndiff --git a/revision.c b/revision.c\nindex de4dce600..72d48a17f 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -175,7 +175,6 @@ static void add_pending_object_with_path(struct rev_info *revs,\n \t\tstrbuf_release(&buf);\n \t\treturn; /* do not add the commit itself */\n \t}\n-\tobj->flags |= USER_GIVEN;\n \tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n }\n \ndiff --git a/revision.h b/revision.h\nindex 5910613cb..83e164039 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -21,9 +21,16 @@\n #define SYMMETRIC_LEFT\t(1u<<8)\n #define PATCHSAME\t(1u<<9)\n #define BOTTOM\t\t(1u<<10)\n-#define USER_GIVEN\t(1u<<25) /* given directly by the user */\n+/*\n+ * Indicates object was reached by traversal. i.e. not given by user on\n+ * command-line or stdin.\n+ * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n+ * filtering trees and blobs, but it may be useful to support filtering commits\n+ * in the future.\n+ */\n+#define NOT_USER_GIVEN\t(1u<<25)\n #define TRACK_LINEAR\t(1u<<26)\n-#define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n+#define ALL_REV_FLAGS\t(((1u<<11)-1) | NOT_USER_GIVEN | TRACK_LINEAR)\n \n #define DECORATE_SHORT_REFS\t1\n #define DECORATE_FULL_REFS\t2\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex c662c97db..11186209b 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -30,6 +30,18 @@ test_expect_success 'verify blob:none omits all 5 blobs' '\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'specify blob explicitly prevents filtering' '\n+\tfile_3=$(git -C r1 ls-files -s file.3 |\n+\t\t awk -f print_2.awk) &&\n+\n+\tfile_4=$(git -C r1 ls-files -s file.4 |\n+\t\t awk -f print_2.awk) &&\n+\n+\tgit -C r1 rev-list --objects --filter=blob:none HEAD $file_3 >observed &&\n+\tgrep -q \"$file_3\" observed &&\n+\ttest_must_fail grep -q \"$file_4\" observed\n+'\n+\n test_expect_success 'verify emitted+omitted == all' '\n \tgit -C r1 rev-list HEAD --objects \\\n \t\t| awk -f print_1.awk \\\n-- \n2.19.0.444.g18242da7ef-goog\n\n"},{"id":"358652","messageId":"a16413f2388ef2d08448e4326c0f010a0b81501b.1537561420.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1537561420.git.matvore@google.com","subject":"[PATCH v9 6/8] list-objects-filter: use BUG rather than die","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-21T20:32:02Z","receivedAt":"2018-09-21T20:32:38Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"In some cases in this file, BUG makes more sense than die. In such\ncases, a we get there from a coding error rather than a user error.\n\n'return' has been removed following some instances of BUG since BUG does\nnot return.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter.c | 11 ++++-------\n 1 file changed, 4 insertions(+), 7 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex a0ba78b20..5f8b1a002 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -44,8 +44,7 @@ static enum list_objects_filter_result filter_blobs_none(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -102,8 +101,7 @@ static enum list_objects_filter_result filter_blobs_limit(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -208,8 +206,7 @@ static enum list_objects_filter_result filter_sparse(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -389,7 +386,7 @@ void *list_objects_filter__init(\n \tassert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n \n \tif (filter_options->choice >= LOFC__COUNT)\n-\t\tdie(\"invalid list-objects filter choice: %d\",\n+\t\tBUG(\"invalid list-objects filter choice: %d\",\n \t\t filter_options->choice);\n \n \tinit_fn = s_filters[filter_options->choice];\n-- \n2.19.0.444.g18242da7ef-goog\n\n"},{"id":"358653","messageId":"4d9bbf5c8af670a969a5b11876a65f0505b1a1b8.1537561420.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1537561420.git.matvore@google.com","subject":"[PATCH v9 7/8] list-objects-filter-options: do not over-strbuf_init","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-21T20:32:03Z","receivedAt":"2018-09-21T20:32:40Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"The function gently_parse_list_objects_filter is either called with\nerrbuf=STRBUF_INIT or errbuf=NULL, but that function calls strbuf_init\nwhen errbuf is not NULL. strbuf_init is only necessary if errbuf\ncontains garbage, and risks a memory leak if errbuf already has a\nnon-STRBUF_INIT state. It should be the caller's responsibility to make\nsure errbuf is not garbage, since garbage content is easily avoidable\nwith STRBUF_INIT.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex c0e2bd6a0..d259bdb2c 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -30,7 +30,6 @@ static int gently_parse_list_objects_filter(\n \n \tif (filter_options->choice) {\n \t\tif (errbuf) {\n-\t\t\tstrbuf_init(errbuf, 0);\n \t\t\tstrbuf_addstr(\n \t\t\t\terrbuf,\n \t\t\t\t_(\"multiple filter-specs cannot be combined\"));\n@@ -71,10 +70,9 @@ static int gently_parse_list_objects_filter(\n \t\treturn 0;\n \t}\n \n-\tif (errbuf) {\n-\t\tstrbuf_init(errbuf, 0);\n+\tif (errbuf)\n \t\tstrbuf_addf(errbuf, \"invalid filter-spec '%s'\", arg);\n-\t}\n+\n \tmemset(filter_options, 0, sizeof(*filter_options));\n \treturn 1;\n }\n-- \n2.19.0.444.g18242da7ef-goog\n\n"},{"id":"358654","messageId":"5e87a9cfcd096784965892c1f87c78bed2f24367.1537561420.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1537561420.git.matvore@google.com","subject":"[PATCH v9 8/8] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-09-21T20:32:04Z","receivedAt":"2018-09-21T20:32:44Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Teach list-objects the \"tree:0\" filter which allows for filtering\nout all tree and blob objects (unless other objects are explicitly\nspecified by the user). The purpose of this patch is to allow smaller\npartial clones.\n\nThe name of this filter - tree:0 - does not explicitly specify that\nit also filters out all blobs, but this should not cause much confusion\nbecause blobs are not at all useful without the trees that refer to\nthem.\n\nI also considered only:commits as a name, but this is inaccurate because\nit suggests that annotated tags are omitted, but actually they are\nincluded.\n\nThe name \"tree:0\" allows later filtering based on depth, i.e. \"tree:1\"\nwould filter out all but the root tree and blobs. In order to avoid\nconfusion between 0 and capital O, the documentation was worded in a\nsomewhat round-about way that also hints at this future improvement to\nthe feature.\n\nSigned-off-by: Matthew DeVore \n---\n Documentation/rev-list-options.txt | 5 +++\n list-objects-filter-options.c | 13 +++++++\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 49 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 28 +++++++++++++++\n t/t5616-partial-clone.sh | 41 +++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 13 +++++++\n 7 files changed, 150 insertions(+)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 7b273635d..5f1672913 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -731,6 +731,11 @@ the requested refs.\n +\n The form '--filter=sparse:path=' similarly uses a sparse-checkout\n specification contained in .\n++\n+The form '--filter=tree:' omits all blobs and trees whose depth\n+from the root tree is >= (minimum depth if an object is located\n+at multiple depths in the commits traversed). Currently, only =0\n+is supported, which omits all blobs and trees.\n \n --no-filter::\n \tTurn off any previous `--filter=` argument.\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex d259bdb2c..e8da2e858 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -49,6 +49,19 @@ static int gently_parse_list_objects_filter(\n \t\t\treturn 0;\n \t\t}\n \n+\t} else if (skip_prefix(arg, \"tree:\", &v0)) {\n+\t\tunsigned long depth;\n+\t\tif (!git_parse_ulong(v0, &depth) || depth != 0) {\n+\t\t\tif (errbuf) {\n+\t\t\t\tstrbuf_addstr(\n+\t\t\t\t\terrbuf,\n+\t\t\t\t\t_(\"only 'tree:0' is supported\"));\n+\t\t\t}\n+\t\t\treturn 1;\n+\t\t}\n+\t\tfilter_options->choice = LOFC_TREE_NONE;\n+\t\treturn 0;\n+\n \t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n \t\tstruct object_context oc;\n \t\tstruct object_id sparse_oid;\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 0000a61f8..af64e5c66 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -10,6 +10,7 @@ enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n+\tLOFC_TREE_NONE,\n \tLOFC_SPARSE_OID,\n \tLOFC_SPARSE_PATH,\n \tLOFC__COUNT /* must be last */\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 5f8b1a002..09b2b05d5 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -79,6 +79,54 @@ static void *filter_blobs_none__init(\n \treturn d;\n }\n \n+/*\n+ * A filter for list-objects to omit ALL trees and blobs from the traversal.\n+ * Can OPTIONALLY collect a list of the omitted OIDs.\n+ */\n+struct filter_trees_none_data {\n+\tstruct oidset *omits;\n+};\n+\n+static enum list_objects_filter_result filter_trees_none(\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_trees_none_data *filter_data = filter_data_;\n+\n+\tswitch (filter_situation) {\n+\tdefault:\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n+\n+\tcase LOFS_BEGIN_TREE:\n+\tcase LOFS_BLOB:\n+\t\tif (filter_data->omits)\n+\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n+\n+\tcase LOFS_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\treturn LOFR_ZERO;\n+\n+\t}\n+}\n+\n+static void* filter_trees_none__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn)\n+{\n+\tstruct filter_trees_none_data *d = xcalloc(1, sizeof(*d));\n+\td->omits = omitted;\n+\n+\t*filter_fn = filter_trees_none;\n+\t*filter_free_fn = free;\n+\treturn d;\n+}\n+\n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n@@ -371,6 +419,7 @@ static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n+\tfilter_trees_none__init,\n \tfilter_sparse_oid__init,\n \tfilter_sparse_path__init,\n };\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 9839b48c1..510d3537f 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -72,6 +72,34 @@ test_expect_success 'get an error for missing tree object' '\n \tgrep -q \"bad tree object\" bad_tree\n '\n \n+test_expect_success 'setup for tests of tree:0' '\n+\tmkdir r1/subtree &&\n+\techo \"This is a file in a subtree\" >r1/subtree/file &&\n+\tgit -C r1 add subtree/file &&\n+\tgit -C r1 commit -m subtree\n+'\n+\n+test_expect_success 'verify tree:0 packfile has no blobs or trees' '\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\t! grep -E \"tree|blob\" objs\n+'\n+\n+test_expect_success 'grab tree directly when using tree:0' '\n+\t# We should get the tree specified directly but not its blobs or subtrees.\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD:\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\tawk \"/tree|blob/{print \\$1}\" objs >trees_and_blobs &&\n+\tgit -C r1 rev-parse HEAD: >expected &&\n+\ttest_cmp expected trees_and_blobs\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex bbbe7537d..7b6294ca5 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -154,6 +154,47 @@ test_expect_success 'partial clone with transfer.fsckobjects=1 uses index-pack -\n \tgrep \"git index-pack.*--fsck-objects\" trace\n '\n \n+test_expect_success 'use fsck before and after manually fetching a missing subtree' '\n+\t# push new commit so server has a subtree\n+\tmkdir src/dir &&\n+\techo \"in dir\" >src/dir/file.txt &&\n+\tgit -C src add dir/file.txt &&\n+\tgit -C src commit -m \"file in dir\" &&\n+\tgit -C src push -u srv master &&\n+\tSUBTREE=$(git -C src rev-parse HEAD:dir) &&\n+\n+\trm -rf dst &&\n+\tgit clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n+\tgit -C dst fsck &&\n+\n+\t# Make sure we only have commits, and all trees and blobs are missing.\n+\tgit -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n+\tawk -f print_1.awk fetched_objects |\n+\txargs -n1 git -C dst cat-file -t >fetched_types &&\n+\n+\tsort -u fetched_types >unique_types.observed &&\n+\techo commit >unique_types.expected &&\n+\ttest_cmp unique_types.expected unique_types.observed &&\n+\n+\t# Auto-fetch a tree with cat-file.\n+\tgit -C dst cat-file -p $SUBTREE >tree_contents &&\n+\tgrep file.txt tree_contents &&\n+\n+\t# fsck still works after an auto-fetch of a tree.\n+\tgit -C dst fsck &&\n+\n+\t# Auto-fetch all remaining trees and blobs with --missing=error\n+\tgit -C dst rev-list master --missing=error --objects >fetched_objects &&\n+\ttest_line_count = 70 fetched_objects &&\n+\n+\tawk -f print_1.awk fetched_objects |\n+\txargs -n1 git -C dst cat-file -t >fetched_types &&\n+\n+\tsort -u fetched_types >unique_types.observed &&\n+\tprintf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n+\ttest_cmp unique_types.expected unique_types.observed\n+'\n+\n test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' '\n \trm -rf src dst &&\n \tgit init src &&\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 11186209b..6e5c41a68 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -224,6 +224,19 @@ test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for tre\n \ttest_must_be_empty rev_list_err\n '\n \n+# Test tree:0 filter.\n+\n+test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n+\tgit -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=tree:0 |\n+\tawk -f print_1.awk |\n+\tsed s/~// |\n+\txargs -n1 git -C r3 cat-file -t |\n+\tsort -u >filtered_types &&\n+\n+\tprintf \"blob\\ntree\\n\" >expected &&\n+\ttest_cmp expected filtered_types\n+'\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.19.0.444.g18242da7ef-goog\n\n"},{"id":"359567","messageId":"CAMfpvhLsrBPtH66bOLX8HKmdKHs6_BpxG4q6P6MUkH-JUafkCg@mail.gmail.com","threadId":"49086","inReplyTo":"d751d56b-84bb-a03d-5f2a-7dbaf8d947cc@jeffhostetler.com","subject":"Re: [PATCH v3 5/5] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-03T19:00:25Z","receivedAt":"2018-10-03T19:00:40Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Tue, Aug 14, 2018 at 8:13 AM Jeff Hostetler wrote:\n>\n> There are a couple of options here:\n> [] If really want to omit all trees and blobs (and we DO NOT want\n> the oidset of everything omitted), then we might be able to\n> shortcut the traversal and speed things up.\n>\n> {} add a LOFR_SKIP_TREE bit to list_objects_filter_result\n> {} test this bit process_tree() and avoid the init_tree_desc() and\n> the while loop and some adjacent setup/tear-down code.\n> {} make this filter something like:\n>\n> case LOFS_BEGIN_TREE:\n> if (filter_data->omits) {\n> oidset_insert(filter_data->omits, &obj->oid);\n> return LOFR_MARK_SEEN; /* ... (hard omit) */\n> } else\n> return LOFR_SKIP_TREE;\n> case LOFS_BLOB:\n> if (filter_data->omits) {\n> oidset_insert(filter_data->omits, &obj->oid);\n> return LOFR_MARK_SEEN; /* ... (hard omit) */\n> else\n> assert(...should not happen...);\n>\n> [] Later, if we choose to actually support a depth>0, we'll probably\n> want a different filter function to conditionally include/exclude\n> blobs, include shallow tree[node]s, and do some of the provisional-\n> omit logic on deep tree[nodes] (in case a tree appears at multiple\n> places/depths in the history). But that can wait.\n>\n> Jeff\n>\n\nJeff, have you made any progress on depth>0 support for the tree\nfilter? I'd like to take a stab at it without duplicating work :)\n\n- Matt\n"},{"id":"359577","messageId":"cover.1538592829.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"[PATCH v10 0/8] filter: support for excluding all trees and blobs","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-03T19:52:11Z","receivedAt":"2018-10-03T19:52:39Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This is a minor change to the previous rollup. It moves positional\narguments to the end of git-rev-list invocations. Here is an interdiff\nfrom v9:\n\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 7b6294ca5..53fbf7db8 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -168,7 +168,8 @@ test_expect_success 'use fsck before and after manually fetching a missing subtr\n \tgit -C dst fsck &&\n \n \t# Make sure we only have commits, and all trees and blobs are missing.\n-\tgit -C dst rev-list master --missing=allow-any --objects >fetched_objects &&\n+\tgit -C dst rev-list --missing=allow-any --objects master \\\n+\t\t>fetched_objects &&\n \tawk -f print_1.awk fetched_objects |\n \txargs -n1 git -C dst cat-file -t >fetched_types &&\n \n@@ -184,7 +185,7 @@ test_expect_success 'use fsck before and after manually fetching a missing subtr\n \tgit -C dst fsck &&\n \n \t# Auto-fetch all remaining trees and blobs with --missing=error\n-\tgit -C dst rev-list master --missing=error --objects >fetched_objects &&\n+\tgit -C dst rev-list --missing=error --objects master >fetched_objects &&\n \ttest_line_count = 70 fetched_objects &&\n \n \tawk -f print_1.awk fetched_objects |\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 6e5c41a68..5a61614b1 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -227,7 +227,8 @@ test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for tre\n # Test tree:0 filter.\n \n test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n-\tgit -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=tree:0 |\n+\tgit -C r3 rev-list --quiet --objects --filter-print-omitted \\\n+\t\t--filter=tree:0 HEAD |\n \tawk -f print_1.awk |\n \tsed s/~// |\n \txargs -n1 git -C r3 cat-file -t |\n\nThank you,\n\nMatthew DeVore (8):\n list-objects: store common func args in struct\n list-objects: refactor to process_tree_contents\n list-objects: always parse trees gently\n rev-list: handle missing tree objects properly\n revision: mark non-user-given objects instead\n list-objects-filter: use BUG rather than die\n list-objects-filter-options: do not over-strbuf_init\n list-objects-filter: implement filter tree:0\n\n Documentation/rev-list-options.txt | 5 +\n builtin/rev-list.c | 11 +-\n list-objects-filter-options.c | 19 +-\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 60 ++++++-\n list-objects.c | 232 +++++++++++++------------\n revision.c | 1 -\n revision.h | 26 ++-\n t/t0410-partial-clone.sh | 45 +++++\n t/t5317-pack-objects-filter-objects.sh | 41 +++++\n t/t5616-partial-clone.sh | 42 +++++\n t/t6112-rev-list-filters-objects.sh | 43 +++++\n 12 files changed, 398 insertions(+), 128 deletions(-)\n\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359578","messageId":"3bff094363c7887bfe6547e520cbe83097cf6f73.1538592829.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1538592829.git.matvore@google.com","subject":"[PATCH v10 1/8] list-objects: store common func args in struct","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-03T19:52:12Z","receivedAt":"2018-10-03T19:52:39Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will make utility functions easier to create, as done by the next\npatch.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 158 +++++++++++++++++++++++--------------------------\n 1 file changed, 74 insertions(+), 84 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex c99c47ac1..584518a3f 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -12,20 +12,25 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n \n-static void process_blob(struct rev_info *revs,\n+struct traversal_context {\n+\tstruct rev_info *revs;\n+\tshow_object_fn show_object;\n+\tshow_commit_fn show_commit;\n+\tvoid *show_data;\n+\tfilter_object_fn filter_fn;\n+\tvoid *filter_data;\n+};\n+\n+static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \n-\tif (!revs->blob_objects)\n+\tif (!ctx->revs->blob_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad blob object\");\n@@ -41,21 +46,21 @@ static void process_blob(struct rev_info *revs,\n \t * may cause the actual filter to report an incomplete list\n \t * of missing objects.\n \t */\n-\tif (revs->exclude_promisor_objects &&\n+\tif (ctx->revs->exclude_promisor_objects &&\n \t !has_object_file(&obj->oid) &&\n \t is_promisor_object(&obj->oid))\n \t\treturn;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BLOB, obj,\n-\t\t\t path->buf, &path->buf[pathlen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n+\t\t\t\t path->buf, &path->buf[pathlen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, path->buf, cb_data);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -81,26 +86,21 @@ static void process_blob(struct rev_info *revs,\n * the link, and how to do it. Whether it necessarily makes\n * any sense what-so-ever to ever do that is another issue.\n */\n-static void process_gitlink(struct rev_info *revs,\n+static void process_gitlink(struct traversal_context *ctx,\n \t\t\t const unsigned char *sha1,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data)\n+\t\t\t const char *name)\n {\n \t/* Nothing to do */\n }\n \n-static void process_tree(struct rev_info *revs,\n+static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &tree->object;\n+\tstruct rev_info *revs = ctx->revs;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n@@ -133,14 +133,14 @@ static void process_tree(struct rev_info *revs,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BEGIN_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, base->buf, cb_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -157,29 +157,25 @@ static void process_tree(struct rev_info *revs,\n \t\t}\n \n \t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(revs,\n+\t\t\tprocess_tree(ctx,\n \t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(revs, entry.oid->hash,\n-\t\t\t\t\tshow, base, entry.path,\n-\t\t\t\t\tcb_data);\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n \t\telse\n-\t\t\tprocess_blob(revs,\n+\t\t\tprocess_blob(ctx,\n \t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t}\n \n-\tif (!(obj->flags & USER_GIVEN) && filter_fn) {\n-\t\tr = filter_fn(LOFS_END_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \t\tif (r & LOFR_MARK_SEEN)\n \t\t\tobj->flags |= SEEN;\n \t\tif (r & LOFR_DO_SHOW)\n-\t\t\tshow(obj, base->buf, cb_data);\n+\t\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \t}\n \n \tstrbuf_setlen(base, baselen);\n@@ -242,19 +238,15 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n-static void traverse_trees_and_blobs(struct rev_info *revs,\n-\t\t\t\t struct strbuf *base,\n-\t\t\t\t show_object_fn show_object,\n-\t\t\t\t void *show_data,\n-\t\t\t\t filter_object_fn filter_fn,\n-\t\t\t\t void *filter_data)\n+static void traverse_trees_and_blobs(struct traversal_context *ctx,\n+\t\t\t\t struct strbuf *base)\n {\n \tint i;\n \n \tassert(base->len == 0);\n \n-\tfor (i = 0; i < revs->pending.nr; i++) {\n-\t\tstruct object_array_entry *pending = revs->pending.objects + i;\n+\tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n+\t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n@@ -262,62 +254,49 @@ static void traverse_trees_and_blobs(struct rev_info *revs,\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tshow_object(obj, name, show_data);\n+\t\t\tctx->show_object(obj, name, ctx->show_data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(revs, (struct tree *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(revs, (struct blob *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n \t\t oid_to_hex(&obj->oid), name);\n \t}\n-\tobject_array_clear(&revs->pending);\n+\tobject_array_clear(&ctx->revs->pending);\n }\n \n-static void do_traverse(struct rev_info *revs,\n-\t\t\tshow_commit_fn show_commit,\n-\t\t\tshow_object_fn show_object,\n-\t\t\tvoid *show_data,\n-\t\t\tfilter_object_fn filter_fn,\n-\t\t\tvoid *filter_data)\n+static void do_traverse(struct traversal_context *ctx)\n {\n \tstruct commit *commit;\n \tstruct strbuf csp; /* callee's scratch pad */\n \tstrbuf_init(&csp, PATH_MAX);\n \n-\twhile ((commit = get_revision(revs)) != NULL) {\n+\twhile ((commit = get_revision(ctx->revs)) != NULL) {\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n \t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(revs, get_commit_tree(commit));\n-\t\tshow_commit(commit, show_data);\n+\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tctx->show_commit(commit, ctx->show_data);\n \n-\t\tif (revs->tree_blobs_in_commit_order)\n+\t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n \t\t\t * NEEDSWORK: Adding the tree and then flushing it here\n \t\t\t * needs a reallocation for each commit. Can we pass the\n \t\t\t * tree directory without allocation churn?\n \t\t\t */\n-\t\t\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t\t\t show_object, show_data,\n-\t\t\t\t\t\t filter_fn, filter_data);\n+\t\t\ttraverse_trees_and_blobs(ctx, &csp);\n \t}\n-\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t show_object, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\ttraverse_trees_and_blobs(ctx, &csp);\n \tstrbuf_release(&csp);\n }\n \n@@ -326,7 +305,14 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t\t show_object_fn show_object,\n \t\t\t void *show_data)\n {\n-\tdo_traverse(revs, show_commit, show_object, show_data, NULL, NULL);\n+\tstruct traversal_context ctx;\n+\tctx.revs = revs;\n+\tctx.show_commit = show_commit;\n+\tctx.show_object = show_object;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\tctx.filter_data = NULL;\n+\tdo_traverse(&ctx);\n }\n \n void traverse_commit_list_filtered(\n@@ -337,14 +323,18 @@ void traverse_commit_list_filtered(\n \tvoid *show_data,\n \tstruct oidset *omitted)\n {\n-\tfilter_object_fn filter_fn = NULL;\n+\tstruct traversal_context ctx;\n \tfilter_free_fn filter_free_fn = NULL;\n-\tvoid *filter_data = NULL;\n-\n-\tfilter_data = list_objects_filter__init(omitted, filter_options,\n-\t\t\t\t\t\t&filter_fn, &filter_free_fn);\n-\tdo_traverse(revs, show_commit, show_object, show_data,\n-\t\t filter_fn, filter_data);\n-\tif (filter_data && filter_free_fn)\n-\t\tfilter_free_fn(filter_data);\n+\n+\tctx.revs = revs;\n+\tctx.show_object = show_object;\n+\tctx.show_commit = show_commit;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\n+\tctx.filter_data = list_objects_filter__init(omitted, filter_options,\n+\t\t\t\t\t\t &ctx.filter_fn, &filter_free_fn);\n+\tdo_traverse(&ctx);\n+\tif (ctx.filter_data && filter_free_fn)\n+\t\tfilter_free_fn(ctx.filter_data);\n }\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359579","messageId":"42ad5623bab52b8207831eab4a4f27e47a318e36.1538592829.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1538592829.git.matvore@google.com","subject":"[PATCH v10 2/8] list-objects: refactor to process_tree_contents","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-03T19:52:13Z","receivedAt":"2018-10-03T19:52:42Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will be used in a follow-up patch to reduce indentation needed when\ninvoking the logic conditionally. i.e. rather than:\n\nif (foo) {\n\twhile (...) {\n\t\t/* this is very indented */\n\t}\n}\n\nwe will have:\n\nif (foo)\n\tprocess_tree_contents(...);\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 68 ++++++++++++++++++++++++++++++--------------------\n 1 file changed, 41 insertions(+), 27 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 584518a3f..ccc529e5e 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -94,6 +94,46 @@ static void process_gitlink(struct traversal_context *ctx,\n \t/* Nothing to do */\n }\n \n+static void process_tree(struct traversal_context *ctx,\n+\t\t\t struct tree *tree,\n+\t\t\t struct strbuf *base,\n+\t\t\t const char *name);\n+\n+static void process_tree_contents(struct traversal_context *ctx,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *base)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tenum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ?\n+\t\tall_entries_interesting : entry_not_interesting;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tif (match != all_entries_interesting) {\n+\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n+\t\t\t\t\t\t &ctx->revs->diffopt.pathspec);\n+\t\t\tif (match == all_entries_not_interesting)\n+\t\t\t\tbreak;\n+\t\t\tif (match == entry_not_interesting)\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (S_ISDIR(entry.mode))\n+\t\t\tprocess_tree(ctx,\n+\t\t\t\t lookup_tree(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t\telse if (S_ISGITLINK(entry.mode))\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash,\n+\t\t\t\t\tbase, entry.path);\n+\t\telse\n+\t\t\tprocess_blob(ctx,\n+\t\t\t\t lookup_blob(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t}\n+}\n+\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n@@ -101,10 +141,6 @@ static void process_tree(struct traversal_context *ctx,\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n-\tstruct tree_desc desc;\n-\tstruct name_entry entry;\n-\tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n-\t\tall_entries_interesting: entry_not_interesting;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \tint gently = revs->ignore_missing_links ||\n@@ -144,29 +180,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tinit_tree_desc(&desc, tree->buffer, tree->size);\n-\n-\twhile (tree_entry(&desc, &entry)) {\n-\t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n-\t\t\t\t\t\t &revs->diffopt.pathspec);\n-\t\t\tif (match == all_entries_not_interesting)\n-\t\t\t\tbreak;\n-\t\t\tif (match == entry_not_interesting)\n-\t\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t}\n+\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359580","messageId":"0d1285a0ec412846e5417ec524a866f6b2c39d82.1538592829.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1538592829.git.matvore@google.com","subject":"[PATCH v10 3/8] list-objects: always parse trees gently","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-03T19:52:14Z","receivedAt":"2018-10-03T19:52:44Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"If parsing fails when revs->ignore_missing_links and\nrevs->exclude_promisor_objects are both false, we print the OID anyway\nin the die(\"bad tree object...\") call, so any message printed by\nparse_tree_gently() is superfluous.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex ccc529e5e..f9b51db7a 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,8 +143,6 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n-\tint gently = revs->ignore_missing_links ||\n-\t\t revs->exclude_promisor_objects;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -152,7 +150,7 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, gently) < 0) {\n+\tif (parse_tree_gently(tree, 1) < 0) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359581","messageId":"a712bad4860f578a339f3d923b1476c11eb4d291.1538592829.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1538592829.git.matvore@google.com","subject":"[PATCH v10 4/8] rev-list: handle missing tree objects properly","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-03T19:52:15Z","receivedAt":"2018-10-03T19:52:48Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Previously, we assumed only blob objects could be missing. This patch\nmakes rev-list handle missing trees like missing blobs. The --missing=*\nand --exclude-promisor-objects flags now work for trees as they already\ndo for blobs. This is demonstrated in t6112.\n\nSigned-off-by: Matthew DeVore \n---\n builtin/rev-list.c | 11 ++++---\n list-objects.c | 11 +++++--\n revision.h | 15 +++++++++\n t/t0410-partial-clone.sh | 45 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 13 ++++++++\n t/t6112-rev-list-filters-objects.sh | 17 ++++++++++\n 6 files changed, 105 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 5b07f3f4a..49d6deed7 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -6,6 +6,7 @@\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"object.h\"\n #include \"object-store.h\"\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n@@ -209,7 +210,8 @@ static inline void finish_object__ma(struct object *obj)\n \t */\n \tswitch (arg_missing_action) {\n \tcase MA_ERROR:\n-\t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n+\t\tdie(\"missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tcase MA_ALLOW_ANY:\n@@ -222,8 +224,8 @@ static inline void finish_object__ma(struct object *obj)\n \tcase MA_ALLOW_PROMISOR:\n \t\tif (is_promisor_object(&obj->oid))\n \t\t\treturn;\n-\t\tdie(\"unexpected missing blob object '%s'\",\n-\t\t oid_to_hex(&obj->oid));\n+\t\tdie(\"unexpected missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tdefault:\n@@ -235,7 +237,7 @@ static inline void finish_object__ma(struct object *obj)\n static int finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\tif (!has_object_file(&obj->oid)) {\n \t\tfinish_object__ma(obj);\n \t\treturn 1;\n \t}\n@@ -373,6 +375,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tinit_revisions(&revs, prefix);\n \trevs.abbrev = DEFAULT_ABBREV;\n \trevs.commit_format = CMIT_FMT_UNSPECIFIED;\n+\trevs.do_not_die_on_missing_tree = 1;\n \n \t/*\n \t * Scan the argument list before invoking setup_revisions(), so that we\ndiff --git a/list-objects.c b/list-objects.c\nindex f9b51db7a..243192af5 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,6 +143,7 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tint failed_parse;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -150,7 +151,9 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, 1) < 0) {\n+\n+\tfailed_parse = parse_tree_gently(tree, 1);\n+\tif (failed_parse) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n@@ -163,7 +166,8 @@ static void process_tree(struct traversal_context *ctx,\n \t\t is_promisor_object(&obj->oid))\n \t\t\treturn;\n \n-\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n+\t\tif (!revs->do_not_die_on_missing_tree)\n+\t\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \n \tstrbuf_addstr(base, name);\n@@ -178,7 +182,8 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tprocess_tree_contents(ctx, tree, base);\n+\tif (!failed_parse)\n+\t\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\ndiff --git a/revision.h b/revision.h\nindex 007278cc1..5910613cb 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -126,6 +126,21 @@ struct rev_info {\n \t\t\tline_level_traverse:1,\n \t\t\ttree_blobs_in_commit_order:1,\n \n+\t\t\t/*\n+\t\t\t * Blobs are shown without regard for their existence.\n+\t\t\t * But not so for trees: unless exclude_promisor_objects\n+\t\t\t * is set and the tree in question is a promisor object;\n+\t\t\t * OR ignore_missing_links is set, the revision walker\n+\t\t\t * dies with a \"bad tree object HASH\" message when\n+\t\t\t * encountering a missing tree. For callers that can\n+\t\t\t * handle missing trees and want them to be filterable\n+\t\t\t * and showable, set this to true. The revision walker\n+\t\t\t * will filter and show such a missing tree as usual,\n+\t\t\t * but will not attempt to recurse into this tree\n+\t\t\t * object.\n+\t\t\t */\n+\t\t\tdo_not_die_on_missing_tree:1,\n+\n \t\t\t/* for internal use only */\n \t\t\texclude_promisor_objects:1;\n \ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 128130066..5bc5b4445 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -186,6 +186,51 @@ test_expect_success 'rev-list stops traversal at missing and promised commit' '\n \t! grep $FOO out\n '\n \n+test_expect_success 'missing tree objects with --missing=allow-promisor and --exclude-promisor-objects' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo foo &&\n+\ttest_commit -C repo bar &&\n+\ttest_commit -C repo baz &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse bar^{tree}) &&\n+\tpromise_and_delete $(git -C repo rev-parse foo^{tree}) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-promisor --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 3 commits, 3 blobs, and 1 tree\n+\ttest_line_count = 7 objs &&\n+\n+\t# Do the same for --exclude-promisor-objects, but with all trees gone.\n+\tpromise_and_delete $(git -C repo rev-parse baz^{tree}) &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 3 commits, no blobs or trees\n+\ttest_line_count = 3 objs\n+'\n+\n+test_expect_success 'missing non-root tree object and rev-list' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\tmkdir repo/dir &&\n+\techo foo >repo/dir/foo &&\n+\tgit -C repo add dir/foo &&\n+\tgit -C repo commit -m \"commit dir/foo\" &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse HEAD:dir) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 1 commit and 1 tree\n+\ttest_line_count = 2 objs\n+'\n+\n test_expect_success 'rev-list stops traversal at missing and promised tree' '\n \trm -rf repo &&\n \ttest_create_repo repo &&\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 6710c8bc8..9839b48c1 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -59,6 +59,19 @@ test_expect_success 'verify normal and blob:none packfiles have same commits/tre\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'get an error for missing tree object' '\n+\tgit init r5 &&\n+\techo foo >r5/foo &&\n+\tgit -C r5 add foo &&\n+\tgit -C r5 commit -m \"foo\" &&\n+\tdel=$(git -C r5 rev-parse HEAD^{tree} | sed \"s|..|&/|\") &&\n+\trm r5/.git/objects/$del &&\n+\ttest_must_fail git -C r5 pack-objects --rev --stdout 2>bad_tree <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgrep -q \"bad tree object\" bad_tree\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex d4ff0b3be..c662c97db 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -195,6 +195,23 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for trees' '\n+\tTREE=$(git -C r3 rev-parse HEAD:dir1) &&\n+\n+\trm r3/.git/objects/$(echo $TREE | sed \"s|^..|&/|\") &&\n+\n+\tgit -C r3 rev-list --quiet --missing=print --objects HEAD >missing_objs 2>rev_list_err &&\n+\techo \"?$TREE\" >expected &&\n+\ttest_cmp expected missing_objs &&\n+\n+\t# do not complain when a missing tree cannot be parsed\n+\ttest_must_be_empty rev_list_err &&\n+\n+\tgit -C r3 rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n+\t! grep $TREE objs &&\n+\ttest_must_be_empty rev_list_err\n+'\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359582","messageId":"6fb22953a334327a48881f691481981af925d56a.1538592829.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1538592829.git.matvore@google.com","subject":"[PATCH v10 5/8] revision: mark non-user-given objects instead","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-03T19:52:16Z","receivedAt":"2018-10-03T19:52:50Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Currently, list-objects.c incorrectly treats all root trees of commits\nas USER_GIVEN. Also, it would be easier to mark objects that are\nnon-user-given instead of user-given, since the places in the code\nwhere we access an object through a reference are more obvious than\nthe places where we access an object that was given by the user.\n\nResolve these two problems by introducing a flag NOT_USER_GIVEN that\nmarks blobs and trees that are non-user-given, replacing USER_GIVEN.\n(Only blobs and trees are marked because this mark is only used when\nfiltering objects, and filtering of other types of objects is not\nsupported yet.)\n\nThis fixes a bug in that git rev-list behaved differently from git\npack-objects. pack-objects would *not* filter objects given explicitly\non the command line and rev-list would filter. This was because the two\ncommands used a different function to add objects to the rev_info\nstruct. This seems to have been an oversight, and pack-objects has the\ncorrect behavior, so I added a test to make sure that rev-list now\nbehaves properly.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 31 +++++++++++++++++------------\n revision.c | 1 -\n revision.h | 11 ++++++++--\n t/t6112-rev-list-filters-objects.sh | 12 +++++++++++\n 4 files changed, 39 insertions(+), 16 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 243192af5..7a1a0929d 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -53,7 +53,7 @@ static void process_blob(struct traversal_context *ctx,\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n \t\t\t\t path->buf, &path->buf[pathlen],\n \t\t\t\t ctx->filter_data);\n@@ -120,17 +120,19 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\tcontinue;\n \t\t}\n \n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\tstruct tree *t = lookup_tree(the_repository, entry.oid);\n+\t\t\tt->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_tree(ctx, t, base, entry.path);\n+\t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid->hash,\n \t\t\t\t\tbase, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\telse {\n+\t\t\tstruct blob *b = lookup_blob(the_repository, entry.oid);\n+\t\t\tb->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_blob(ctx, b, base, entry.path);\n+\t\t}\n \t}\n }\n \n@@ -171,7 +173,7 @@ static void process_tree(struct traversal_context *ctx,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -185,7 +187,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (!failed_parse)\n \t\tprocess_tree_contents(ctx, tree, base);\n \n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -301,8 +303,11 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n-\t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tif (get_commit_tree(commit)) {\n+\t\t\tstruct tree *tree = get_commit_tree(commit);\n+\t\t\ttree->object.flags |= NOT_USER_GIVEN;\n+\t\t\tadd_pending_tree(ctx->revs, tree);\n+\t\t}\n \t\tctx->show_commit(commit, ctx->show_data);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\ndiff --git a/revision.c b/revision.c\nindex de4dce600..72d48a17f 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -175,7 +175,6 @@ static void add_pending_object_with_path(struct rev_info *revs,\n \t\tstrbuf_release(&buf);\n \t\treturn; /* do not add the commit itself */\n \t}\n-\tobj->flags |= USER_GIVEN;\n \tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n }\n \ndiff --git a/revision.h b/revision.h\nindex 5910613cb..83e164039 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -21,9 +21,16 @@\n #define SYMMETRIC_LEFT\t(1u<<8)\n #define PATCHSAME\t(1u<<9)\n #define BOTTOM\t\t(1u<<10)\n-#define USER_GIVEN\t(1u<<25) /* given directly by the user */\n+/*\n+ * Indicates object was reached by traversal. i.e. not given by user on\n+ * command-line or stdin.\n+ * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n+ * filtering trees and blobs, but it may be useful to support filtering commits\n+ * in the future.\n+ */\n+#define NOT_USER_GIVEN\t(1u<<25)\n #define TRACK_LINEAR\t(1u<<26)\n-#define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n+#define ALL_REV_FLAGS\t(((1u<<11)-1) | NOT_USER_GIVEN | TRACK_LINEAR)\n \n #define DECORATE_SHORT_REFS\t1\n #define DECORATE_FULL_REFS\t2\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex c662c97db..11186209b 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -30,6 +30,18 @@ test_expect_success 'verify blob:none omits all 5 blobs' '\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'specify blob explicitly prevents filtering' '\n+\tfile_3=$(git -C r1 ls-files -s file.3 |\n+\t\t awk -f print_2.awk) &&\n+\n+\tfile_4=$(git -C r1 ls-files -s file.4 |\n+\t\t awk -f print_2.awk) &&\n+\n+\tgit -C r1 rev-list --objects --filter=blob:none HEAD $file_3 >observed &&\n+\tgrep -q \"$file_3\" observed &&\n+\ttest_must_fail grep -q \"$file_4\" observed\n+'\n+\n test_expect_success 'verify emitted+omitted == all' '\n \tgit -C r1 rev-list HEAD --objects \\\n \t\t| awk -f print_1.awk \\\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359583","messageId":"a16413f2388ef2d08448e4326c0f010a0b81501b.1538592829.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1538592829.git.matvore@google.com","subject":"[PATCH v10 6/8] list-objects-filter: use BUG rather than die","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-03T19:52:17Z","receivedAt":"2018-10-03T19:52:52Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"In some cases in this file, BUG makes more sense than die. In such\ncases, a we get there from a coding error rather than a user error.\n\n'return' has been removed following some instances of BUG since BUG does\nnot return.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter.c | 11 ++++-------\n 1 file changed, 4 insertions(+), 7 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex a0ba78b20..5f8b1a002 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -44,8 +44,7 @@ static enum list_objects_filter_result filter_blobs_none(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -102,8 +101,7 @@ static enum list_objects_filter_result filter_blobs_limit(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -208,8 +206,7 @@ static enum list_objects_filter_result filter_sparse(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -389,7 +386,7 @@ void *list_objects_filter__init(\n \tassert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n \n \tif (filter_options->choice >= LOFC__COUNT)\n-\t\tdie(\"invalid list-objects filter choice: %d\",\n+\t\tBUG(\"invalid list-objects filter choice: %d\",\n \t\t filter_options->choice);\n \n \tinit_fn = s_filters[filter_options->choice];\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359584","messageId":"4d9bbf5c8af670a969a5b11876a65f0505b1a1b8.1538592829.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1538592829.git.matvore@google.com","subject":"[PATCH v10 7/8] list-objects-filter-options: do not over-strbuf_init","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-03T19:52:18Z","receivedAt":"2018-10-03T19:52:54Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"The function gently_parse_list_objects_filter is either called with\nerrbuf=STRBUF_INIT or errbuf=NULL, but that function calls strbuf_init\nwhen errbuf is not NULL. strbuf_init is only necessary if errbuf\ncontains garbage, and risks a memory leak if errbuf already has a\nnon-STRBUF_INIT state. It should be the caller's responsibility to make\nsure errbuf is not garbage, since garbage content is easily avoidable\nwith STRBUF_INIT.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex c0e2bd6a0..d259bdb2c 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -30,7 +30,6 @@ static int gently_parse_list_objects_filter(\n \n \tif (filter_options->choice) {\n \t\tif (errbuf) {\n-\t\t\tstrbuf_init(errbuf, 0);\n \t\t\tstrbuf_addstr(\n \t\t\t\terrbuf,\n \t\t\t\t_(\"multiple filter-specs cannot be combined\"));\n@@ -71,10 +70,9 @@ static int gently_parse_list_objects_filter(\n \t\treturn 0;\n \t}\n \n-\tif (errbuf) {\n-\t\tstrbuf_init(errbuf, 0);\n+\tif (errbuf)\n \t\tstrbuf_addf(errbuf, \"invalid filter-spec '%s'\", arg);\n-\t}\n+\n \tmemset(filter_options, 0, sizeof(*filter_options));\n \treturn 1;\n }\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359585","messageId":"e59bf8cc5aaf2ab1e1f3e28033e5335b18489bdf.1538592829.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1538592829.git.matvore@google.com","subject":"[PATCH v10 8/8] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-03T19:52:19Z","receivedAt":"2018-10-03T19:52:57Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Teach list-objects the \"tree:0\" filter which allows for filtering\nout all tree and blob objects (unless other objects are explicitly\nspecified by the user). The purpose of this patch is to allow smaller\npartial clones.\n\nThe name of this filter - tree:0 - does not explicitly specify that\nit also filters out all blobs, but this should not cause much confusion\nbecause blobs are not at all useful without the trees that refer to\nthem.\n\nI also considered only:commits as a name, but this is inaccurate because\nit suggests that annotated tags are omitted, but actually they are\nincluded.\n\nThe name \"tree:0\" allows later filtering based on depth, i.e. \"tree:1\"\nwould filter out all but the root tree and blobs. In order to avoid\nconfusion between 0 and capital O, the documentation was worded in a\nsomewhat round-about way that also hints at this future improvement to\nthe feature.\n\nSigned-off-by: Matthew DeVore \n---\n Documentation/rev-list-options.txt | 5 +++\n list-objects-filter-options.c | 13 +++++++\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 49 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 28 +++++++++++++++\n t/t5616-partial-clone.sh | 42 ++++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 14 ++++++++\n 7 files changed, 152 insertions(+)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 7b273635d..5f1672913 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -731,6 +731,11 @@ the requested refs.\n +\n The form '--filter=sparse:path=' similarly uses a sparse-checkout\n specification contained in .\n++\n+The form '--filter=tree:' omits all blobs and trees whose depth\n+from the root tree is >= (minimum depth if an object is located\n+at multiple depths in the commits traversed). Currently, only =0\n+is supported, which omits all blobs and trees.\n \n --no-filter::\n \tTurn off any previous `--filter=` argument.\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex d259bdb2c..e8da2e858 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -49,6 +49,19 @@ static int gently_parse_list_objects_filter(\n \t\t\treturn 0;\n \t\t}\n \n+\t} else if (skip_prefix(arg, \"tree:\", &v0)) {\n+\t\tunsigned long depth;\n+\t\tif (!git_parse_ulong(v0, &depth) || depth != 0) {\n+\t\t\tif (errbuf) {\n+\t\t\t\tstrbuf_addstr(\n+\t\t\t\t\terrbuf,\n+\t\t\t\t\t_(\"only 'tree:0' is supported\"));\n+\t\t\t}\n+\t\t\treturn 1;\n+\t\t}\n+\t\tfilter_options->choice = LOFC_TREE_NONE;\n+\t\treturn 0;\n+\n \t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n \t\tstruct object_context oc;\n \t\tstruct object_id sparse_oid;\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 0000a61f8..af64e5c66 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -10,6 +10,7 @@ enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n+\tLOFC_TREE_NONE,\n \tLOFC_SPARSE_OID,\n \tLOFC_SPARSE_PATH,\n \tLOFC__COUNT /* must be last */\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 5f8b1a002..09b2b05d5 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -79,6 +79,54 @@ static void *filter_blobs_none__init(\n \treturn d;\n }\n \n+/*\n+ * A filter for list-objects to omit ALL trees and blobs from the traversal.\n+ * Can OPTIONALLY collect a list of the omitted OIDs.\n+ */\n+struct filter_trees_none_data {\n+\tstruct oidset *omits;\n+};\n+\n+static enum list_objects_filter_result filter_trees_none(\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_trees_none_data *filter_data = filter_data_;\n+\n+\tswitch (filter_situation) {\n+\tdefault:\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n+\n+\tcase LOFS_BEGIN_TREE:\n+\tcase LOFS_BLOB:\n+\t\tif (filter_data->omits)\n+\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n+\n+\tcase LOFS_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\treturn LOFR_ZERO;\n+\n+\t}\n+}\n+\n+static void* filter_trees_none__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn)\n+{\n+\tstruct filter_trees_none_data *d = xcalloc(1, sizeof(*d));\n+\td->omits = omitted;\n+\n+\t*filter_fn = filter_trees_none;\n+\t*filter_free_fn = free;\n+\treturn d;\n+}\n+\n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n@@ -371,6 +419,7 @@ static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n+\tfilter_trees_none__init,\n \tfilter_sparse_oid__init,\n \tfilter_sparse_path__init,\n };\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 9839b48c1..510d3537f 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -72,6 +72,34 @@ test_expect_success 'get an error for missing tree object' '\n \tgrep -q \"bad tree object\" bad_tree\n '\n \n+test_expect_success 'setup for tests of tree:0' '\n+\tmkdir r1/subtree &&\n+\techo \"This is a file in a subtree\" >r1/subtree/file &&\n+\tgit -C r1 add subtree/file &&\n+\tgit -C r1 commit -m subtree\n+'\n+\n+test_expect_success 'verify tree:0 packfile has no blobs or trees' '\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\t! grep -E \"tree|blob\" objs\n+'\n+\n+test_expect_success 'grab tree directly when using tree:0' '\n+\t# We should get the tree specified directly but not its blobs or subtrees.\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD:\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\tawk \"/tree|blob/{print \\$1}\" objs >trees_and_blobs &&\n+\tgit -C r1 rev-parse HEAD: >expected &&\n+\ttest_cmp expected trees_and_blobs\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex bbbe7537d..53fbf7db8 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -154,6 +154,48 @@ test_expect_success 'partial clone with transfer.fsckobjects=1 uses index-pack -\n \tgrep \"git index-pack.*--fsck-objects\" trace\n '\n \n+test_expect_success 'use fsck before and after manually fetching a missing subtree' '\n+\t# push new commit so server has a subtree\n+\tmkdir src/dir &&\n+\techo \"in dir\" >src/dir/file.txt &&\n+\tgit -C src add dir/file.txt &&\n+\tgit -C src commit -m \"file in dir\" &&\n+\tgit -C src push -u srv master &&\n+\tSUBTREE=$(git -C src rev-parse HEAD:dir) &&\n+\n+\trm -rf dst &&\n+\tgit clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n+\tgit -C dst fsck &&\n+\n+\t# Make sure we only have commits, and all trees and blobs are missing.\n+\tgit -C dst rev-list --missing=allow-any --objects master \\\n+\t\t>fetched_objects &&\n+\tawk -f print_1.awk fetched_objects |\n+\txargs -n1 git -C dst cat-file -t >fetched_types &&\n+\n+\tsort -u fetched_types >unique_types.observed &&\n+\techo commit >unique_types.expected &&\n+\ttest_cmp unique_types.expected unique_types.observed &&\n+\n+\t# Auto-fetch a tree with cat-file.\n+\tgit -C dst cat-file -p $SUBTREE >tree_contents &&\n+\tgrep file.txt tree_contents &&\n+\n+\t# fsck still works after an auto-fetch of a tree.\n+\tgit -C dst fsck &&\n+\n+\t# Auto-fetch all remaining trees and blobs with --missing=error\n+\tgit -C dst rev-list --missing=error --objects master >fetched_objects &&\n+\ttest_line_count = 70 fetched_objects &&\n+\n+\tawk -f print_1.awk fetched_objects |\n+\txargs -n1 git -C dst cat-file -t >fetched_types &&\n+\n+\tsort -u fetched_types >unique_types.observed &&\n+\tprintf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n+\ttest_cmp unique_types.expected unique_types.observed\n+'\n+\n test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' '\n \trm -rf src dst &&\n \tgit init src &&\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 11186209b..5a61614b1 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -224,6 +224,20 @@ test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for tre\n \ttest_must_be_empty rev_list_err\n '\n \n+# Test tree:0 filter.\n+\n+test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n+\tgit -C r3 rev-list --quiet --objects --filter-print-omitted \\\n+\t\t--filter=tree:0 HEAD |\n+\tawk -f print_1.awk |\n+\tsed s/~// |\n+\txargs -n1 git -C r3 cat-file -t |\n+\tsort -u >filtered_types &&\n+\n+\tprintf \"blob\\ntree\\n\" >expected &&\n+\ttest_cmp expected filtered_types\n+'\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359598","messageId":"CAMfpvhLmXyk5MKoe=zC_sVgURwcxLBWL3zZWkrncGeL0FcyQDQ@mail.gmail.com","threadId":"49086","inReplyTo":"cover.1538592829.git.matvore@google.com","subject":"Re: [PATCH v10 0/8] filter: support for excluding all trees and blobs","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-03T23:08:06Z","receivedAt":"2018-10-03T23:08:21Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Wed, Oct 3, 2018 at 12:52 PM Matthew DeVore wrote:\n>\n> This is a minor change to the previous rollup. It moves positional\n> arguments to the end of git-rev-list invocations. Here is an interdiff\n> from v9:\n...\nThere is another problem with this patchset related to dropped exit\ncodes and pipelines. In t6112, we run \"git cat-file -t\" on an object\nwith was rm'd without being promised. It was printing an error and\ngoing undetected because it was upstream in a pipeline. The file was\nremoved in the previous test.\n\nSo I fixed the previous test to clone the repository before\nmanipulating it, and I fixed the latter test to not mask Git exit\ncodes :) (This is a really insidious pattern and I should have taken\nit more seriously.) Below is an interdiff. The two tests are added in\ndifferent commits, so each commit had to be fixed up.\n\nI'll send a re-roll in two days or so if there are no more comments.\n\ndiff --git a/t/t6112-rev-list-filters-objects.sh\nb/t/t6112-rev-list-filters-objects.sh\nindex 5a61614b1..c8e3d87c4 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -210,16 +210,21 @@ test_expect_success 'verify sparse:oid=oid-ish\nomits top-level files' '\n test_expect_success 'rev-list W/ --missing=print and\n--missing=allow-any for trees' '\n TREE=$(git -C r3 rev-parse HEAD:dir1) &&\n\n- rm r3/.git/objects/$(echo $TREE | sed \"s|^..|&/|\") &&\n+ # Create a spare repo because we will be deleting objects\nfrom this one.\n+ git clone r3 r3.b &&\n\n- git -C r3 rev-list --quiet --missing=print --objects HEAD\n>missing_objs 2>rev_list_err &&\n+ rm r3.b/.git/objects/$(echo $TREE | sed \"s|^..|&/|\") &&\n+\n+ git -C r3.b rev-list --quiet --missing=print --objects HEAD \\\n+ >missing_objs 2>rev_list_err &&\n echo \"?$TREE\" >expected &&\n test_cmp expected missing_objs &&\n\n # do not complain when a missing tree cannot be parsed\n test_must_be_empty rev_list_err &&\n\n- git -C r3 rev-list --missing=allow-any --objects HEAD >objs\n2>rev_list_err &&\n+ git -C r3.b rev-list --missing=allow-any --objects HEAD \\\n+ >objs 2>rev_list_err &&\n ! grep $TREE objs &&\n test_must_be_empty rev_list_err\n '\n@@ -228,12 +233,13 @@ test_expect_success 'rev-list W/ --missing=print\nand --missing=allow-any for tre\n\n test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n git -C r3 rev-list --quiet --objects --filter-print-omitted \\\n- --filter=tree:0 HEAD |\n- awk -f print_1.awk |\n+ --filter=tree:0 HEAD >revs &&\n+\n+ awk -f print_1.awk revs |\n sed s/~// |\n- xargs -n1 git -C r3 cat-file -t |\n- sort -u >filtered_types &&\n+ xargs -n1 git -C r3 cat-file -t >unsorted_filtered_types &&\n\n+ sort -u unsorted_filtered_types >filtered_types &&\n printf \"blob\\ntree\\n\" >expected &&\n test_cmp expected filtered_types\n '\n"},{"id":"359740","messageId":"cover.1538774738.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"[PATCH v11 0/8] filter: support for excluding all trees and blobs","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-05T21:31:19Z","receivedAt":"2018-10-05T21:31:39Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Here is a clean re-rollup fixing the issue I found earlier. It fixes a problem\nstemming from a discarded exit code, which masked a crash in Git. The crash was\nnot a bug because an earlier test deleted a loose object file. The fix was to\nmake that test manipulate a clone rather than the original repo.\n\nAn interdiff from v10 is below.\n\nThank you,\nMatt\n\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 5a61614b1..c8e3d87c4 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -210,16 +210,21 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for trees' '\n TREE=$(git -C r3 rev-parse HEAD:dir1) &&\n \n- rm r3/.git/objects/$(echo $TREE | sed \"s|^..|&/|\") &&\n+ # Create a spare repo because we will be deleting objects from this one.\n+ git clone r3 r3.b &&\n \n- git -C r3 rev-list --quiet --missing=print --objects HEAD >missing_objs 2>rev_list_err &&\n+ rm r3.b/.git/objects/$(echo $TREE | sed \"s|^..|&/|\") &&\n+\n+ git -C r3.b rev-list --quiet --missing=print --objects HEAD \\\n+ >missing_objs 2>rev_list_err &&\n echo \"?$TREE\" >expected &&\n test_cmp expected missing_objs &&\n \n # do not complain when a missing tree cannot be parsed\n test_must_be_empty rev_list_err &&\n \n- git -C r3 rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n+ git -C r3.b rev-list --missing=allow-any --objects HEAD \\\n+ >objs 2>rev_list_err &&\n ! grep $TREE objs &&\n test_must_be_empty rev_list_err\n '\n@@ -228,12 +233,13 @@ test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for tre\n \n test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n git -C r3 rev-list --quiet --objects --filter-print-omitted \\\n- --filter=tree:0 HEAD |\n- awk -f print_1.awk |\n+ --filter=tree:0 HEAD >revs &&\n+\n+ awk -f print_1.awk revs |\n sed s/~// |\n- xargs -n1 git -C r3 cat-file -t |\n- sort -u >filtered_types &&\n+ xargs -n1 git -C r3 cat-file -t >unsorted_filtered_types &&\n \n+ sort -u unsorted_filtered_types >filtered_types &&\n printf \"blob\\ntree\\n\" >expected &&\n test_cmp expected filtered_types\n '\n\nMatthew DeVore (8):\n list-objects: store common func args in struct\n list-objects: refactor to process_tree_contents\n list-objects: always parse trees gently\n rev-list: handle missing tree objects properly\n revision: mark non-user-given objects instead\n list-objects-filter: use BUG rather than die\n list-objects-filter-options: do not over-strbuf_init\n list-objects-filter: implement filter tree:0\n\n Documentation/rev-list-options.txt | 5 +\n builtin/rev-list.c | 11 +-\n list-objects-filter-options.c | 19 +-\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 60 ++++++-\n list-objects.c | 232 +++++++++++++------------\n revision.c | 1 -\n revision.h | 26 ++-\n t/t0410-partial-clone.sh | 45 +++++\n t/t5317-pack-objects-filter-objects.sh | 41 +++++\n t/t5616-partial-clone.sh | 42 +++++\n t/t6112-rev-list-filters-objects.sh | 49 ++++++\n 12 files changed, 404 insertions(+), 128 deletions(-)\n\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359741","messageId":"3bff094363c7887bfe6547e520cbe83097cf6f73.1538774738.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1538774738.git.matvore@google.com","subject":"[PATCH v11 1/8] list-objects: store common func args in struct","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-05T21:31:20Z","receivedAt":"2018-10-05T21:31:40Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will make utility functions easier to create, as done by the next\npatch.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 158 +++++++++++++++++++++++--------------------------\n 1 file changed, 74 insertions(+), 84 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex c99c47ac1..584518a3f 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -12,20 +12,25 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n \n-static void process_blob(struct rev_info *revs,\n+struct traversal_context {\n+\tstruct rev_info *revs;\n+\tshow_object_fn show_object;\n+\tshow_commit_fn show_commit;\n+\tvoid *show_data;\n+\tfilter_object_fn filter_fn;\n+\tvoid *filter_data;\n+};\n+\n+static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \n-\tif (!revs->blob_objects)\n+\tif (!ctx->revs->blob_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad blob object\");\n@@ -41,21 +46,21 @@ static void process_blob(struct rev_info *revs,\n \t * may cause the actual filter to report an incomplete list\n \t * of missing objects.\n \t */\n-\tif (revs->exclude_promisor_objects &&\n+\tif (ctx->revs->exclude_promisor_objects &&\n \t !has_object_file(&obj->oid) &&\n \t is_promisor_object(&obj->oid))\n \t\treturn;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BLOB, obj,\n-\t\t\t path->buf, &path->buf[pathlen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n+\t\t\t\t path->buf, &path->buf[pathlen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, path->buf, cb_data);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -81,26 +86,21 @@ static void process_blob(struct rev_info *revs,\n * the link, and how to do it. Whether it necessarily makes\n * any sense what-so-ever to ever do that is another issue.\n */\n-static void process_gitlink(struct rev_info *revs,\n+static void process_gitlink(struct traversal_context *ctx,\n \t\t\t const unsigned char *sha1,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data)\n+\t\t\t const char *name)\n {\n \t/* Nothing to do */\n }\n \n-static void process_tree(struct rev_info *revs,\n+static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &tree->object;\n+\tstruct rev_info *revs = ctx->revs;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n@@ -133,14 +133,14 @@ static void process_tree(struct rev_info *revs,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BEGIN_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, base->buf, cb_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -157,29 +157,25 @@ static void process_tree(struct rev_info *revs,\n \t\t}\n \n \t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(revs,\n+\t\t\tprocess_tree(ctx,\n \t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(revs, entry.oid->hash,\n-\t\t\t\t\tshow, base, entry.path,\n-\t\t\t\t\tcb_data);\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n \t\telse\n-\t\t\tprocess_blob(revs,\n+\t\t\tprocess_blob(ctx,\n \t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t}\n \n-\tif (!(obj->flags & USER_GIVEN) && filter_fn) {\n-\t\tr = filter_fn(LOFS_END_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \t\tif (r & LOFR_MARK_SEEN)\n \t\t\tobj->flags |= SEEN;\n \t\tif (r & LOFR_DO_SHOW)\n-\t\t\tshow(obj, base->buf, cb_data);\n+\t\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \t}\n \n \tstrbuf_setlen(base, baselen);\n@@ -242,19 +238,15 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n-static void traverse_trees_and_blobs(struct rev_info *revs,\n-\t\t\t\t struct strbuf *base,\n-\t\t\t\t show_object_fn show_object,\n-\t\t\t\t void *show_data,\n-\t\t\t\t filter_object_fn filter_fn,\n-\t\t\t\t void *filter_data)\n+static void traverse_trees_and_blobs(struct traversal_context *ctx,\n+\t\t\t\t struct strbuf *base)\n {\n \tint i;\n \n \tassert(base->len == 0);\n \n-\tfor (i = 0; i < revs->pending.nr; i++) {\n-\t\tstruct object_array_entry *pending = revs->pending.objects + i;\n+\tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n+\t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n@@ -262,62 +254,49 @@ static void traverse_trees_and_blobs(struct rev_info *revs,\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tshow_object(obj, name, show_data);\n+\t\t\tctx->show_object(obj, name, ctx->show_data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(revs, (struct tree *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(revs, (struct blob *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n \t\t oid_to_hex(&obj->oid), name);\n \t}\n-\tobject_array_clear(&revs->pending);\n+\tobject_array_clear(&ctx->revs->pending);\n }\n \n-static void do_traverse(struct rev_info *revs,\n-\t\t\tshow_commit_fn show_commit,\n-\t\t\tshow_object_fn show_object,\n-\t\t\tvoid *show_data,\n-\t\t\tfilter_object_fn filter_fn,\n-\t\t\tvoid *filter_data)\n+static void do_traverse(struct traversal_context *ctx)\n {\n \tstruct commit *commit;\n \tstruct strbuf csp; /* callee's scratch pad */\n \tstrbuf_init(&csp, PATH_MAX);\n \n-\twhile ((commit = get_revision(revs)) != NULL) {\n+\twhile ((commit = get_revision(ctx->revs)) != NULL) {\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n \t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(revs, get_commit_tree(commit));\n-\t\tshow_commit(commit, show_data);\n+\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tctx->show_commit(commit, ctx->show_data);\n \n-\t\tif (revs->tree_blobs_in_commit_order)\n+\t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n \t\t\t * NEEDSWORK: Adding the tree and then flushing it here\n \t\t\t * needs a reallocation for each commit. Can we pass the\n \t\t\t * tree directory without allocation churn?\n \t\t\t */\n-\t\t\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t\t\t show_object, show_data,\n-\t\t\t\t\t\t filter_fn, filter_data);\n+\t\t\ttraverse_trees_and_blobs(ctx, &csp);\n \t}\n-\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t show_object, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\ttraverse_trees_and_blobs(ctx, &csp);\n \tstrbuf_release(&csp);\n }\n \n@@ -326,7 +305,14 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t\t show_object_fn show_object,\n \t\t\t void *show_data)\n {\n-\tdo_traverse(revs, show_commit, show_object, show_data, NULL, NULL);\n+\tstruct traversal_context ctx;\n+\tctx.revs = revs;\n+\tctx.show_commit = show_commit;\n+\tctx.show_object = show_object;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\tctx.filter_data = NULL;\n+\tdo_traverse(&ctx);\n }\n \n void traverse_commit_list_filtered(\n@@ -337,14 +323,18 @@ void traverse_commit_list_filtered(\n \tvoid *show_data,\n \tstruct oidset *omitted)\n {\n-\tfilter_object_fn filter_fn = NULL;\n+\tstruct traversal_context ctx;\n \tfilter_free_fn filter_free_fn = NULL;\n-\tvoid *filter_data = NULL;\n-\n-\tfilter_data = list_objects_filter__init(omitted, filter_options,\n-\t\t\t\t\t\t&filter_fn, &filter_free_fn);\n-\tdo_traverse(revs, show_commit, show_object, show_data,\n-\t\t filter_fn, filter_data);\n-\tif (filter_data && filter_free_fn)\n-\t\tfilter_free_fn(filter_data);\n+\n+\tctx.revs = revs;\n+\tctx.show_object = show_object;\n+\tctx.show_commit = show_commit;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\n+\tctx.filter_data = list_objects_filter__init(omitted, filter_options,\n+\t\t\t\t\t\t &ctx.filter_fn, &filter_free_fn);\n+\tdo_traverse(&ctx);\n+\tif (ctx.filter_data && filter_free_fn)\n+\t\tfilter_free_fn(ctx.filter_data);\n }\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359742","messageId":"42ad5623bab52b8207831eab4a4f27e47a318e36.1538774738.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1538774738.git.matvore@google.com","subject":"[PATCH v11 2/8] list-objects: refactor to process_tree_contents","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-05T21:31:21Z","receivedAt":"2018-10-05T21:31:43Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will be used in a follow-up patch to reduce indentation needed when\ninvoking the logic conditionally. i.e. rather than:\n\nif (foo) {\n\twhile (...) {\n\t\t/* this is very indented */\n\t}\n}\n\nwe will have:\n\nif (foo)\n\tprocess_tree_contents(...);\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 68 ++++++++++++++++++++++++++++++--------------------\n 1 file changed, 41 insertions(+), 27 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 584518a3f..ccc529e5e 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -94,6 +94,46 @@ static void process_gitlink(struct traversal_context *ctx,\n \t/* Nothing to do */\n }\n \n+static void process_tree(struct traversal_context *ctx,\n+\t\t\t struct tree *tree,\n+\t\t\t struct strbuf *base,\n+\t\t\t const char *name);\n+\n+static void process_tree_contents(struct traversal_context *ctx,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *base)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tenum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ?\n+\t\tall_entries_interesting : entry_not_interesting;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tif (match != all_entries_interesting) {\n+\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n+\t\t\t\t\t\t &ctx->revs->diffopt.pathspec);\n+\t\t\tif (match == all_entries_not_interesting)\n+\t\t\t\tbreak;\n+\t\t\tif (match == entry_not_interesting)\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (S_ISDIR(entry.mode))\n+\t\t\tprocess_tree(ctx,\n+\t\t\t\t lookup_tree(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t\telse if (S_ISGITLINK(entry.mode))\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash,\n+\t\t\t\t\tbase, entry.path);\n+\t\telse\n+\t\t\tprocess_blob(ctx,\n+\t\t\t\t lookup_blob(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t}\n+}\n+\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n@@ -101,10 +141,6 @@ static void process_tree(struct traversal_context *ctx,\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n-\tstruct tree_desc desc;\n-\tstruct name_entry entry;\n-\tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n-\t\tall_entries_interesting: entry_not_interesting;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \tint gently = revs->ignore_missing_links ||\n@@ -144,29 +180,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tinit_tree_desc(&desc, tree->buffer, tree->size);\n-\n-\twhile (tree_entry(&desc, &entry)) {\n-\t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n-\t\t\t\t\t\t &revs->diffopt.pathspec);\n-\t\t\tif (match == all_entries_not_interesting)\n-\t\t\t\tbreak;\n-\t\t\tif (match == entry_not_interesting)\n-\t\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t}\n+\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359743","messageId":"0d1285a0ec412846e5417ec524a866f6b2c39d82.1538774738.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1538774738.git.matvore@google.com","subject":"[PATCH v11 3/8] list-objects: always parse trees gently","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-05T21:31:22Z","receivedAt":"2018-10-05T21:31:44Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"If parsing fails when revs->ignore_missing_links and\nrevs->exclude_promisor_objects are both false, we print the OID anyway\nin the die(\"bad tree object...\") call, so any message printed by\nparse_tree_gently() is superfluous.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex ccc529e5e..f9b51db7a 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,8 +143,6 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n-\tint gently = revs->ignore_missing_links ||\n-\t\t revs->exclude_promisor_objects;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -152,7 +150,7 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, gently) < 0) {\n+\tif (parse_tree_gently(tree, 1) < 0) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359744","messageId":"793739b7df9c7f46e7b2e6a49640b44266415924.1538774738.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1538774738.git.matvore@google.com","subject":"[PATCH v11 4/8] rev-list: handle missing tree objects properly","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-05T21:31:23Z","receivedAt":"2018-10-05T21:31:48Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Previously, we assumed only blob objects could be missing. This patch\nmakes rev-list handle missing trees like missing blobs. The --missing=*\nand --exclude-promisor-objects flags now work for trees as they already\ndo for blobs. This is demonstrated in t6112.\n\nSigned-off-by: Matthew DeVore \n---\n builtin/rev-list.c | 11 ++++---\n list-objects.c | 11 +++++--\n revision.h | 15 +++++++++\n t/t0410-partial-clone.sh | 45 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 13 ++++++++\n t/t6112-rev-list-filters-objects.sh | 22 +++++++++++++\n 6 files changed, 110 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 5b07f3f4a..49d6deed7 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -6,6 +6,7 @@\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"object.h\"\n #include \"object-store.h\"\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n@@ -209,7 +210,8 @@ static inline void finish_object__ma(struct object *obj)\n \t */\n \tswitch (arg_missing_action) {\n \tcase MA_ERROR:\n-\t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n+\t\tdie(\"missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tcase MA_ALLOW_ANY:\n@@ -222,8 +224,8 @@ static inline void finish_object__ma(struct object *obj)\n \tcase MA_ALLOW_PROMISOR:\n \t\tif (is_promisor_object(&obj->oid))\n \t\t\treturn;\n-\t\tdie(\"unexpected missing blob object '%s'\",\n-\t\t oid_to_hex(&obj->oid));\n+\t\tdie(\"unexpected missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tdefault:\n@@ -235,7 +237,7 @@ static inline void finish_object__ma(struct object *obj)\n static int finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\tif (!has_object_file(&obj->oid)) {\n \t\tfinish_object__ma(obj);\n \t\treturn 1;\n \t}\n@@ -373,6 +375,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tinit_revisions(&revs, prefix);\n \trevs.abbrev = DEFAULT_ABBREV;\n \trevs.commit_format = CMIT_FMT_UNSPECIFIED;\n+\trevs.do_not_die_on_missing_tree = 1;\n \n \t/*\n \t * Scan the argument list before invoking setup_revisions(), so that we\ndiff --git a/list-objects.c b/list-objects.c\nindex f9b51db7a..243192af5 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,6 +143,7 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tint failed_parse;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -150,7 +151,9 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, 1) < 0) {\n+\n+\tfailed_parse = parse_tree_gently(tree, 1);\n+\tif (failed_parse) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n@@ -163,7 +166,8 @@ static void process_tree(struct traversal_context *ctx,\n \t\t is_promisor_object(&obj->oid))\n \t\t\treturn;\n \n-\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n+\t\tif (!revs->do_not_die_on_missing_tree)\n+\t\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \n \tstrbuf_addstr(base, name);\n@@ -178,7 +182,8 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tprocess_tree_contents(ctx, tree, base);\n+\tif (!failed_parse)\n+\t\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\ndiff --git a/revision.h b/revision.h\nindex 007278cc1..5910613cb 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -126,6 +126,21 @@ struct rev_info {\n \t\t\tline_level_traverse:1,\n \t\t\ttree_blobs_in_commit_order:1,\n \n+\t\t\t/*\n+\t\t\t * Blobs are shown without regard for their existence.\n+\t\t\t * But not so for trees: unless exclude_promisor_objects\n+\t\t\t * is set and the tree in question is a promisor object;\n+\t\t\t * OR ignore_missing_links is set, the revision walker\n+\t\t\t * dies with a \"bad tree object HASH\" message when\n+\t\t\t * encountering a missing tree. For callers that can\n+\t\t\t * handle missing trees and want them to be filterable\n+\t\t\t * and showable, set this to true. The revision walker\n+\t\t\t * will filter and show such a missing tree as usual,\n+\t\t\t * but will not attempt to recurse into this tree\n+\t\t\t * object.\n+\t\t\t */\n+\t\t\tdo_not_die_on_missing_tree:1,\n+\n \t\t\t/* for internal use only */\n \t\t\texclude_promisor_objects:1;\n \ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 128130066..5bc5b4445 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -186,6 +186,51 @@ test_expect_success 'rev-list stops traversal at missing and promised commit' '\n \t! grep $FOO out\n '\n \n+test_expect_success 'missing tree objects with --missing=allow-promisor and --exclude-promisor-objects' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo foo &&\n+\ttest_commit -C repo bar &&\n+\ttest_commit -C repo baz &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse bar^{tree}) &&\n+\tpromise_and_delete $(git -C repo rev-parse foo^{tree}) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-promisor --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 3 commits, 3 blobs, and 1 tree\n+\ttest_line_count = 7 objs &&\n+\n+\t# Do the same for --exclude-promisor-objects, but with all trees gone.\n+\tpromise_and_delete $(git -C repo rev-parse baz^{tree}) &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 3 commits, no blobs or trees\n+\ttest_line_count = 3 objs\n+'\n+\n+test_expect_success 'missing non-root tree object and rev-list' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\tmkdir repo/dir &&\n+\techo foo >repo/dir/foo &&\n+\tgit -C repo add dir/foo &&\n+\tgit -C repo commit -m \"commit dir/foo\" &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse HEAD:dir) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 1 commit and 1 tree\n+\ttest_line_count = 2 objs\n+'\n+\n test_expect_success 'rev-list stops traversal at missing and promised tree' '\n \trm -rf repo &&\n \ttest_create_repo repo &&\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 6710c8bc8..9839b48c1 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -59,6 +59,19 @@ test_expect_success 'verify normal and blob:none packfiles have same commits/tre\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'get an error for missing tree object' '\n+\tgit init r5 &&\n+\techo foo >r5/foo &&\n+\tgit -C r5 add foo &&\n+\tgit -C r5 commit -m \"foo\" &&\n+\tdel=$(git -C r5 rev-parse HEAD^{tree} | sed \"s|..|&/|\") &&\n+\trm r5/.git/objects/$del &&\n+\ttest_must_fail git -C r5 pack-objects --rev --stdout 2>bad_tree <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgrep -q \"bad tree object\" bad_tree\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex d4ff0b3be..efe5a2467 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -195,6 +195,28 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for trees' '\n+\tTREE=$(git -C r3 rev-parse HEAD:dir1) &&\n+\n+\t# Create a spare repo because we will be deleting objects from this one.\n+\tgit clone r3 r3.b &&\n+\n+\trm r3.b/.git/objects/$(echo $TREE | sed \"s|^..|&/|\") &&\n+\n+\tgit -C r3.b rev-list --quiet --missing=print --objects HEAD \\\n+\t\t>missing_objs 2>rev_list_err &&\n+\techo \"?$TREE\" >expected &&\n+\ttest_cmp expected missing_objs &&\n+\n+\t# do not complain when a missing tree cannot be parsed\n+\ttest_must_be_empty rev_list_err &&\n+\n+\tgit -C r3.b rev-list --missing=allow-any --objects HEAD \\\n+\t\t>objs 2>rev_list_err &&\n+\t! grep $TREE objs &&\n+\ttest_must_be_empty rev_list_err\n+'\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359745","messageId":"9ef4d3ada1f9ce89abd529c83137625469eaf920.1538774738.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1538774738.git.matvore@google.com","subject":"[PATCH v11 5/8] revision: mark non-user-given objects instead","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-05T21:31:24Z","receivedAt":"2018-10-05T21:31:50Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Currently, list-objects.c incorrectly treats all root trees of commits\nas USER_GIVEN. Also, it would be easier to mark objects that are\nnon-user-given instead of user-given, since the places in the code\nwhere we access an object through a reference are more obvious than\nthe places where we access an object that was given by the user.\n\nResolve these two problems by introducing a flag NOT_USER_GIVEN that\nmarks blobs and trees that are non-user-given, replacing USER_GIVEN.\n(Only blobs and trees are marked because this mark is only used when\nfiltering objects, and filtering of other types of objects is not\nsupported yet.)\n\nThis fixes a bug in that git rev-list behaved differently from git\npack-objects. pack-objects would *not* filter objects given explicitly\non the command line and rev-list would filter. This was because the two\ncommands used a different function to add objects to the rev_info\nstruct. This seems to have been an oversight, and pack-objects has the\ncorrect behavior, so I added a test to make sure that rev-list now\nbehaves properly.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 31 +++++++++++++++++------------\n revision.c | 1 -\n revision.h | 11 ++++++++--\n t/t6112-rev-list-filters-objects.sh | 12 +++++++++++\n 4 files changed, 39 insertions(+), 16 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 243192af5..7a1a0929d 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -53,7 +53,7 @@ static void process_blob(struct traversal_context *ctx,\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n \t\t\t\t path->buf, &path->buf[pathlen],\n \t\t\t\t ctx->filter_data);\n@@ -120,17 +120,19 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\tcontinue;\n \t\t}\n \n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\tstruct tree *t = lookup_tree(the_repository, entry.oid);\n+\t\t\tt->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_tree(ctx, t, base, entry.path);\n+\t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid->hash,\n \t\t\t\t\tbase, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\telse {\n+\t\t\tstruct blob *b = lookup_blob(the_repository, entry.oid);\n+\t\t\tb->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_blob(ctx, b, base, entry.path);\n+\t\t}\n \t}\n }\n \n@@ -171,7 +173,7 @@ static void process_tree(struct traversal_context *ctx,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -185,7 +187,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (!failed_parse)\n \t\tprocess_tree_contents(ctx, tree, base);\n \n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -301,8 +303,11 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n-\t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tif (get_commit_tree(commit)) {\n+\t\t\tstruct tree *tree = get_commit_tree(commit);\n+\t\t\ttree->object.flags |= NOT_USER_GIVEN;\n+\t\t\tadd_pending_tree(ctx->revs, tree);\n+\t\t}\n \t\tctx->show_commit(commit, ctx->show_data);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\ndiff --git a/revision.c b/revision.c\nindex de4dce600..72d48a17f 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -175,7 +175,6 @@ static void add_pending_object_with_path(struct rev_info *revs,\n \t\tstrbuf_release(&buf);\n \t\treturn; /* do not add the commit itself */\n \t}\n-\tobj->flags |= USER_GIVEN;\n \tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n }\n \ndiff --git a/revision.h b/revision.h\nindex 5910613cb..83e164039 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -21,9 +21,16 @@\n #define SYMMETRIC_LEFT\t(1u<<8)\n #define PATCHSAME\t(1u<<9)\n #define BOTTOM\t\t(1u<<10)\n-#define USER_GIVEN\t(1u<<25) /* given directly by the user */\n+/*\n+ * Indicates object was reached by traversal. i.e. not given by user on\n+ * command-line or stdin.\n+ * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n+ * filtering trees and blobs, but it may be useful to support filtering commits\n+ * in the future.\n+ */\n+#define NOT_USER_GIVEN\t(1u<<25)\n #define TRACK_LINEAR\t(1u<<26)\n-#define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n+#define ALL_REV_FLAGS\t(((1u<<11)-1) | NOT_USER_GIVEN | TRACK_LINEAR)\n \n #define DECORATE_SHORT_REFS\t1\n #define DECORATE_FULL_REFS\t2\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex efe5a2467..ccbc64413 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -30,6 +30,18 @@ test_expect_success 'verify blob:none omits all 5 blobs' '\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'specify blob explicitly prevents filtering' '\n+\tfile_3=$(git -C r1 ls-files -s file.3 |\n+\t\t awk -f print_2.awk) &&\n+\n+\tfile_4=$(git -C r1 ls-files -s file.4 |\n+\t\t awk -f print_2.awk) &&\n+\n+\tgit -C r1 rev-list --objects --filter=blob:none HEAD $file_3 >observed &&\n+\tgrep -q \"$file_3\" observed &&\n+\ttest_must_fail grep -q \"$file_4\" observed\n+'\n+\n test_expect_success 'verify emitted+omitted == all' '\n \tgit -C r1 rev-list HEAD --objects \\\n \t\t| awk -f print_1.awk \\\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359746","messageId":"5b337881c3b6cf0f7a9483e578f613f59c609c44.1538774738.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1538774738.git.matvore@google.com","subject":"[PATCH v11 6/8] list-objects-filter: use BUG rather than die","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-05T21:31:25Z","receivedAt":"2018-10-05T21:31:52Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"In some cases in this file, BUG makes more sense than die. In such\ncases, a we get there from a coding error rather than a user error.\n\n'return' has been removed following some instances of BUG since BUG does\nnot return.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter.c | 11 ++++-------\n 1 file changed, 4 insertions(+), 7 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex a0ba78b20..5f8b1a002 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -44,8 +44,7 @@ static enum list_objects_filter_result filter_blobs_none(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -102,8 +101,7 @@ static enum list_objects_filter_result filter_blobs_limit(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -208,8 +206,7 @@ static enum list_objects_filter_result filter_sparse(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -389,7 +386,7 @@ void *list_objects_filter__init(\n \tassert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n \n \tif (filter_options->choice >= LOFC__COUNT)\n-\t\tdie(\"invalid list-objects filter choice: %d\",\n+\t\tBUG(\"invalid list-objects filter choice: %d\",\n \t\t filter_options->choice);\n \n \tinit_fn = s_filters[filter_options->choice];\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359747","messageId":"1a055a109adadaca6eff89514089fbde739d600c.1538774738.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1538774738.git.matvore@google.com","subject":"[PATCH v11 7/8] list-objects-filter-options: do not over-strbuf_init","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-05T21:31:26Z","receivedAt":"2018-10-05T21:31:54Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"The function gently_parse_list_objects_filter is either called with\nerrbuf=STRBUF_INIT or errbuf=NULL, but that function calls strbuf_init\nwhen errbuf is not NULL. strbuf_init is only necessary if errbuf\ncontains garbage, and risks a memory leak if errbuf already has a\nnon-STRBUF_INIT state. It should be the caller's responsibility to make\nsure errbuf is not garbage, since garbage content is easily avoidable\nwith STRBUF_INIT.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex c0e2bd6a0..d259bdb2c 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -30,7 +30,6 @@ static int gently_parse_list_objects_filter(\n \n \tif (filter_options->choice) {\n \t\tif (errbuf) {\n-\t\t\tstrbuf_init(errbuf, 0);\n \t\t\tstrbuf_addstr(\n \t\t\t\terrbuf,\n \t\t\t\t_(\"multiple filter-specs cannot be combined\"));\n@@ -71,10 +70,9 @@ static int gently_parse_list_objects_filter(\n \t\treturn 0;\n \t}\n \n-\tif (errbuf) {\n-\t\tstrbuf_init(errbuf, 0);\n+\tif (errbuf)\n \t\tstrbuf_addf(errbuf, \"invalid filter-spec '%s'\", arg);\n-\t}\n+\n \tmemset(filter_options, 0, sizeof(*filter_options));\n \treturn 1;\n }\n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359748","messageId":"6ca50a28e292f32127ea706a2aef39f834ac7702.1538774738.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1538774738.git.matvore@google.com","subject":"[PATCH v11 8/8] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-05T21:31:27Z","receivedAt":"2018-10-05T21:31:57Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Teach list-objects the \"tree:0\" filter which allows for filtering\nout all tree and blob objects (unless other objects are explicitly\nspecified by the user). The purpose of this patch is to allow smaller\npartial clones.\n\nThe name of this filter - tree:0 - does not explicitly specify that\nit also filters out all blobs, but this should not cause much confusion\nbecause blobs are not at all useful without the trees that refer to\nthem.\n\nI also considered only:commits as a name, but this is inaccurate because\nit suggests that annotated tags are omitted, but actually they are\nincluded.\n\nThe name \"tree:0\" allows later filtering based on depth, i.e. \"tree:1\"\nwould filter out all but the root tree and blobs. In order to avoid\nconfusion between 0 and capital O, the documentation was worded in a\nsomewhat round-about way that also hints at this future improvement to\nthe feature.\n\nSigned-off-by: Matthew DeVore \n---\n Documentation/rev-list-options.txt | 5 +++\n list-objects-filter-options.c | 13 +++++++\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 49 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 28 +++++++++++++++\n t/t5616-partial-clone.sh | 42 ++++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 15 ++++++++\n 7 files changed, 153 insertions(+)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 7b273635d..5f1672913 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -731,6 +731,11 @@ the requested refs.\n +\n The form '--filter=sparse:path=' similarly uses a sparse-checkout\n specification contained in .\n++\n+The form '--filter=tree:' omits all blobs and trees whose depth\n+from the root tree is >= (minimum depth if an object is located\n+at multiple depths in the commits traversed). Currently, only =0\n+is supported, which omits all blobs and trees.\n \n --no-filter::\n \tTurn off any previous `--filter=` argument.\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex d259bdb2c..e8da2e858 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -49,6 +49,19 @@ static int gently_parse_list_objects_filter(\n \t\t\treturn 0;\n \t\t}\n \n+\t} else if (skip_prefix(arg, \"tree:\", &v0)) {\n+\t\tunsigned long depth;\n+\t\tif (!git_parse_ulong(v0, &depth) || depth != 0) {\n+\t\t\tif (errbuf) {\n+\t\t\t\tstrbuf_addstr(\n+\t\t\t\t\terrbuf,\n+\t\t\t\t\t_(\"only 'tree:0' is supported\"));\n+\t\t\t}\n+\t\t\treturn 1;\n+\t\t}\n+\t\tfilter_options->choice = LOFC_TREE_NONE;\n+\t\treturn 0;\n+\n \t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n \t\tstruct object_context oc;\n \t\tstruct object_id sparse_oid;\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 0000a61f8..af64e5c66 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -10,6 +10,7 @@ enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n+\tLOFC_TREE_NONE,\n \tLOFC_SPARSE_OID,\n \tLOFC_SPARSE_PATH,\n \tLOFC__COUNT /* must be last */\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 5f8b1a002..09b2b05d5 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -79,6 +79,54 @@ static void *filter_blobs_none__init(\n \treturn d;\n }\n \n+/*\n+ * A filter for list-objects to omit ALL trees and blobs from the traversal.\n+ * Can OPTIONALLY collect a list of the omitted OIDs.\n+ */\n+struct filter_trees_none_data {\n+\tstruct oidset *omits;\n+};\n+\n+static enum list_objects_filter_result filter_trees_none(\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_trees_none_data *filter_data = filter_data_;\n+\n+\tswitch (filter_situation) {\n+\tdefault:\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n+\n+\tcase LOFS_BEGIN_TREE:\n+\tcase LOFS_BLOB:\n+\t\tif (filter_data->omits)\n+\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n+\n+\tcase LOFS_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\treturn LOFR_ZERO;\n+\n+\t}\n+}\n+\n+static void* filter_trees_none__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn)\n+{\n+\tstruct filter_trees_none_data *d = xcalloc(1, sizeof(*d));\n+\td->omits = omitted;\n+\n+\t*filter_fn = filter_trees_none;\n+\t*filter_free_fn = free;\n+\treturn d;\n+}\n+\n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n@@ -371,6 +419,7 @@ static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n+\tfilter_trees_none__init,\n \tfilter_sparse_oid__init,\n \tfilter_sparse_path__init,\n };\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 9839b48c1..510d3537f 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -72,6 +72,34 @@ test_expect_success 'get an error for missing tree object' '\n \tgrep -q \"bad tree object\" bad_tree\n '\n \n+test_expect_success 'setup for tests of tree:0' '\n+\tmkdir r1/subtree &&\n+\techo \"This is a file in a subtree\" >r1/subtree/file &&\n+\tgit -C r1 add subtree/file &&\n+\tgit -C r1 commit -m subtree\n+'\n+\n+test_expect_success 'verify tree:0 packfile has no blobs or trees' '\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\t! grep -E \"tree|blob\" objs\n+'\n+\n+test_expect_success 'grab tree directly when using tree:0' '\n+\t# We should get the tree specified directly but not its blobs or subtrees.\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD:\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\tawk \"/tree|blob/{print \\$1}\" objs >trees_and_blobs &&\n+\tgit -C r1 rev-parse HEAD: >expected &&\n+\ttest_cmp expected trees_and_blobs\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex bbbe7537d..53fbf7db8 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -154,6 +154,48 @@ test_expect_success 'partial clone with transfer.fsckobjects=1 uses index-pack -\n \tgrep \"git index-pack.*--fsck-objects\" trace\n '\n \n+test_expect_success 'use fsck before and after manually fetching a missing subtree' '\n+\t# push new commit so server has a subtree\n+\tmkdir src/dir &&\n+\techo \"in dir\" >src/dir/file.txt &&\n+\tgit -C src add dir/file.txt &&\n+\tgit -C src commit -m \"file in dir\" &&\n+\tgit -C src push -u srv master &&\n+\tSUBTREE=$(git -C src rev-parse HEAD:dir) &&\n+\n+\trm -rf dst &&\n+\tgit clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n+\tgit -C dst fsck &&\n+\n+\t# Make sure we only have commits, and all trees and blobs are missing.\n+\tgit -C dst rev-list --missing=allow-any --objects master \\\n+\t\t>fetched_objects &&\n+\tawk -f print_1.awk fetched_objects |\n+\txargs -n1 git -C dst cat-file -t >fetched_types &&\n+\n+\tsort -u fetched_types >unique_types.observed &&\n+\techo commit >unique_types.expected &&\n+\ttest_cmp unique_types.expected unique_types.observed &&\n+\n+\t# Auto-fetch a tree with cat-file.\n+\tgit -C dst cat-file -p $SUBTREE >tree_contents &&\n+\tgrep file.txt tree_contents &&\n+\n+\t# fsck still works after an auto-fetch of a tree.\n+\tgit -C dst fsck &&\n+\n+\t# Auto-fetch all remaining trees and blobs with --missing=error\n+\tgit -C dst rev-list --missing=error --objects master >fetched_objects &&\n+\ttest_line_count = 70 fetched_objects &&\n+\n+\tawk -f print_1.awk fetched_objects |\n+\txargs -n1 git -C dst cat-file -t >fetched_types &&\n+\n+\tsort -u fetched_types >unique_types.observed &&\n+\tprintf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n+\ttest_cmp unique_types.expected unique_types.observed\n+'\n+\n test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' '\n \trm -rf src dst &&\n \tgit init src &&\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex ccbc64413..c8e3d87c4 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -229,6 +229,21 @@ test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for tre\n \ttest_must_be_empty rev_list_err\n '\n \n+# Test tree:0 filter.\n+\n+test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n+\tgit -C r3 rev-list --quiet --objects --filter-print-omitted \\\n+\t\t--filter=tree:0 HEAD >revs &&\n+\n+\tawk -f print_1.awk revs |\n+\tsed s/~// |\n+\txargs -n1 git -C r3 cat-file -t >unsorted_filtered_types &&\n+\n+\tsort -u unsorted_filtered_types >filtered_types &&\n+\tprintf \"blob\\ntree\\n\" >expected &&\n+\ttest_cmp expected filtered_types\n+'\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.19.0.605.g01d371f741-goog\n\n"},{"id":"359780","messageId":"xmqqpnwmvcow.fsf@gitster-ct.c.googlers.com","threadId":"49086","inReplyTo":"6ca50a28e292f32127ea706a2aef39f834ac7702.1538774738.git.matvore@google.com","subject":"Re: [PATCH v11 8/8] list-objects-filter: implement filter tree:0","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-07T00:10:39Z","receivedAt":"2018-10-07T00:10:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthew DeVore writes:\n\n> The name \"tree:0\" allows later filtering based on depth, i.e. \"tree:1\"\n> would filter out all but the root tree and blobs. In order to avoid\n> confusion between 0 and capital O, the documentation was worded in a\n> somewhat round-about way that also hints at this future improvement to\n> the feature.\n>\n> Signed-off-by: Matthew DeVore \n\nThanks.\n\n> diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\n> index 7b273635d..5f1672913 100644\n> --- a/Documentation/rev-list-options.txt\n> +++ b/Documentation/rev-list-options.txt\n> @@ -731,6 +731,11 @@ the requested refs.\n> +\n> The form '--filter=sparse:path=' similarly uses a sparse-checkout\n> specification contained in .\n> ++\n> +The form '--filter=tree:' omits all blobs and trees whose depth\n> +from the root tree is >= (minimum depth if an object is located\n> +at multiple depths in the commits traversed). Currently, only =0\n> +is supported, which omits all blobs and trees.\n\nOK.\n\n> diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\n> index 9839b48c1..510d3537f 100755\n> --- a/t/t5317-pack-objects-filter-objects.sh\n> +++ b/t/t5317-pack-objects-filter-objects.sh\n> @@ -72,6 +72,34 @@ test_expect_success 'get an error for missing tree object' '\n> \tgrep -q \"bad tree object\" bad_tree\n> '\n\nAs output made inside test_expect_{succcess,failure} are discarded\nby default and shown while debugging tests, there is no strong\nreason to use \"grep -q\" in our tests. I saw a few instances of\n\"grep -q\" added in this series including this one\n\n\ttest_must_fail grep -q \"$file_4\" observed\n\nthat should probably be\n\n\t! grep \"$file_4\" observed\n\n> +\tprintf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n> ...\n> +\tprintf \"blob\\ntree\\n\" >expected &&\n\nUsing test_write_lines is probably easier to read.\n\nOther than these two minor classes of nits, the series look quite\nwell cooked to me.\n\nThanks.\n"},{"id":"359840","messageId":"CAMfpvhJi+T+zRS8pkZkN_x5xYD3gxTjeczm18VLYWtxfb4Zvmw@mail.gmail.com","threadId":"49086","inReplyTo":"xmqqpnwmvcow.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v11 8/8] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-08T17:23:36Z","receivedAt":"2018-10-08T17:23:51Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Sat, Oct 6, 2018 at 5:10 PM Junio C Hamano wrote:\n>\n> As output made inside test_expect_{succcess,failure} are discarded\n> by default and shown while debugging tests, there is no strong\n> reason to use \"grep -q\" in our tests. I saw a few instances of\n> \"grep -q\" added in this series including this one\n>\n> test_must_fail grep -q \"$file_4\" observed\n>\n> that should probably be\n>\n> ! grep \"$file_4\" observed\nYeah, I remember I read in the testing guidelines that you should just\nuse ! for non-Git commands since it's not our job to make sure these\ntools are not crashing. Thank you for pointing this out.\n\n>\n> > + printf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n> > ...\n> > + printf \"blob\\ntree\\n\" >expected &&\n>\n> Using test_write_lines is probably easier to read.\n\nDone. Below is an interdiff. Let me know if you want a reroll soon.\nOtherwise, I will send one later this week.\n\n- Matt\n\ndiff --git a/t/t5317-pack-objects-filter-objects.sh\nb/t/t5317-pack-objects-filter-objects.sh\nindex 510d3537f..d9dccf4d4 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -69,7 +69,7 @@ test_expect_success 'get an error for missing tree object' '\n test_must_fail git -C r5 pack-objects --rev --stdout\n2>bad_tree <<-EOF &&\n HEAD\n EOF\n- grep -q \"bad tree object\" bad_tree\n+ grep \"bad tree object\" bad_tree\n '\n\n test_expect_success 'setup for tests of tree:0' '\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 53fbf7db8..392caa08f 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -192,7 +192,7 @@ test_expect_success 'use fsck before and after\nmanually fetching a missing subtr\n xargs -n1 git -C dst cat-file -t >fetched_types &&\n\n sort -u fetched_types >unique_types.observed &&\n- printf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n+ test_write_lines blob commit tree >unique_types.expected &&\n test_cmp unique_types.expected unique_types.observed\n '\n\ndiff --git a/t/t6112-rev-list-filters-objects.sh\nb/t/t6112-rev-list-filters-objects.sh\nindex c8e3d87c4..08e0c7db6 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -38,8 +38,8 @@ test_expect_success 'specify blob explicitly\nprevents filtering' '\n awk -f print_2.awk) &&\n\n git -C r1 rev-list --objects --filter=blob:none HEAD $file_3\n>observed &&\n- grep -q \"$file_3\" observed &&\n- test_must_fail grep -q \"$file_4\" observed\n+ grep \"$file_3\" observed &&\n+ ! grep \"$file_4\" observed\n '\n\n test_expect_success 'verify emitted+omitted == all' '\n@@ -240,7 +240,7 @@ test_expect_success 'verify tree:0 includes trees\nin \"filtered\" output' '\n xargs -n1 git -C r3 cat-file -t >unsorted_filtered_types &&\n\n sort -u unsorted_filtered_types >filtered_types &&\n- printf \"blob\\ntree\\n\" >expected &&\n+ test_write_lines blob tree >expected &&\n test_cmp expected filtered_types\n '\n"},{"id":"360340","messageId":"cover.1539373969.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1533854545.git.matvore@google.com","subject":"[PATCH v12 0/8] filter: support for excluding all trees and blobs","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-12T20:01:41Z","receivedAt":"2018-10-12T20:02:02Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Here is a re-roll-up since I haven't received any additional corrections for\nalmost a week. The changes are very slight and just for clean-up so it is ready\nto be promoted to master.\n\nThis is the interdiff from last time:\n\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 510d3537f..d9dccf4d4 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -69,7 +69,7 @@ test_expect_success 'get an error for missing tree object' '\n test_must_fail git -C r5 pack-objects --rev --stdout 2>bad_tree <<-EOF &&\n HEAD\n EOF\n- grep -q \"bad tree object\" bad_tree\n+ grep \"bad tree object\" bad_tree\n '\n \n test_expect_success 'setup for tests of tree:0' '\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 53fbf7db8..392caa08f 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -192,7 +192,7 @@ test_expect_success 'use fsck before and after manually fetching a missing subtr\n xargs -n1 git -C dst cat-file -t >fetched_types &&\n \n sort -u fetched_types >unique_types.observed &&\n- printf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n+ test_write_lines blob commit tree >unique_types.expected &&\n test_cmp unique_types.expected unique_types.observed\n '\n \ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex c8e3d87c4..08e0c7db6 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -38,8 +38,8 @@ test_expect_success 'specify blob explicitly prevents filtering' '\n awk -f print_2.awk) &&\n \n git -C r1 rev-list --objects --filter=blob:none HEAD $file_3 >observed &&\n- grep -q \"$file_3\" observed &&\n- test_must_fail grep -q \"$file_4\" observed\n+ grep \"$file_3\" observed &&\n+ ! grep \"$file_4\" observed\n '\n \n test_expect_success 'verify emitted+omitted == all' '\n@@ -240,7 +240,7 @@ test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n xargs -n1 git -C r3 cat-file -t >unsorted_filtered_types &&\n \n sort -u unsorted_filtered_types >filtered_types &&\n- printf \"blob\\ntree\\n\" >expected &&\n+ test_write_lines blob tree >expected &&\n test_cmp expected filtered_types\n '\n \n\nThanks,\n\nMatthew DeVore (8):\n list-objects: store common func args in struct\n list-objects: refactor to process_tree_contents\n list-objects: always parse trees gently\n rev-list: handle missing tree objects properly\n revision: mark non-user-given objects instead\n list-objects-filter: use BUG rather than die\n list-objects-filter-options: do not over-strbuf_init\n list-objects-filter: implement filter tree:0\n\n Documentation/rev-list-options.txt | 5 +\n builtin/rev-list.c | 11 +-\n list-objects-filter-options.c | 19 +-\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 60 ++++++-\n list-objects.c | 232 +++++++++++++------------\n revision.c | 1 -\n revision.h | 26 ++-\n t/t0410-partial-clone.sh | 45 +++++\n t/t5317-pack-objects-filter-objects.sh | 41 +++++\n t/t5616-partial-clone.sh | 42 +++++\n t/t6112-rev-list-filters-objects.sh | 49 ++++++\n 12 files changed, 404 insertions(+), 128 deletions(-)\n\n-- \n2.19.1.331.ge82ca0e54c-goog\n\n"},{"id":"360341","messageId":"3bff094363c7887bfe6547e520cbe83097cf6f73.1539373969.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1539373969.git.matvore@google.com","subject":"[PATCH v12 1/8] list-objects: store common func args in struct","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-12T20:01:42Z","receivedAt":"2018-10-12T20:02:13Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will make utility functions easier to create, as done by the next\npatch.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 158 +++++++++++++++++++++++--------------------------\n 1 file changed, 74 insertions(+), 84 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex c99c47ac1..584518a3f 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -12,20 +12,25 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n \n-static void process_blob(struct rev_info *revs,\n+struct traversal_context {\n+\tstruct rev_info *revs;\n+\tshow_object_fn show_object;\n+\tshow_commit_fn show_commit;\n+\tvoid *show_data;\n+\tfilter_object_fn filter_fn;\n+\tvoid *filter_data;\n+};\n+\n+static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \n-\tif (!revs->blob_objects)\n+\tif (!ctx->revs->blob_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad blob object\");\n@@ -41,21 +46,21 @@ static void process_blob(struct rev_info *revs,\n \t * may cause the actual filter to report an incomplete list\n \t * of missing objects.\n \t */\n-\tif (revs->exclude_promisor_objects &&\n+\tif (ctx->revs->exclude_promisor_objects &&\n \t !has_object_file(&obj->oid) &&\n \t is_promisor_object(&obj->oid))\n \t\treturn;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BLOB, obj,\n-\t\t\t path->buf, &path->buf[pathlen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n+\t\t\t\t path->buf, &path->buf[pathlen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, path->buf, cb_data);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -81,26 +86,21 @@ static void process_blob(struct rev_info *revs,\n * the link, and how to do it. Whether it necessarily makes\n * any sense what-so-ever to ever do that is another issue.\n */\n-static void process_gitlink(struct rev_info *revs,\n+static void process_gitlink(struct traversal_context *ctx,\n \t\t\t const unsigned char *sha1,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data)\n+\t\t\t const char *name)\n {\n \t/* Nothing to do */\n }\n \n-static void process_tree(struct rev_info *revs,\n+static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n-\t\t\t show_object_fn show,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name,\n-\t\t\t void *cb_data,\n-\t\t\t filter_object_fn filter_fn,\n-\t\t\t void *filter_data)\n+\t\t\t const char *name)\n {\n \tstruct object *obj = &tree->object;\n+\tstruct rev_info *revs = ctx->revs;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n@@ -133,14 +133,14 @@ static void process_tree(struct rev_info *revs,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && filter_fn)\n-\t\tr = filter_fn(LOFS_BEGIN_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tshow(obj, base->buf, cb_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -157,29 +157,25 @@ static void process_tree(struct rev_info *revs,\n \t\t}\n \n \t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(revs,\n+\t\t\tprocess_tree(ctx,\n \t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(revs, entry.oid->hash,\n-\t\t\t\t\tshow, base, entry.path,\n-\t\t\t\t\tcb_data);\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n \t\telse\n-\t\t\tprocess_blob(revs,\n+\t\t\tprocess_blob(ctx,\n \t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data, filter_fn, filter_data);\n+\t\t\t\t base, entry.path);\n \t}\n \n-\tif (!(obj->flags & USER_GIVEN) && filter_fn) {\n-\t\tr = filter_fn(LOFS_END_TREE, obj,\n-\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t filter_data);\n+\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n+\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t ctx->filter_data);\n \t\tif (r & LOFR_MARK_SEEN)\n \t\t\tobj->flags |= SEEN;\n \t\tif (r & LOFR_DO_SHOW)\n-\t\t\tshow(obj, base->buf, cb_data);\n+\t\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \t}\n \n \tstrbuf_setlen(base, baselen);\n@@ -242,19 +238,15 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n-static void traverse_trees_and_blobs(struct rev_info *revs,\n-\t\t\t\t struct strbuf *base,\n-\t\t\t\t show_object_fn show_object,\n-\t\t\t\t void *show_data,\n-\t\t\t\t filter_object_fn filter_fn,\n-\t\t\t\t void *filter_data)\n+static void traverse_trees_and_blobs(struct traversal_context *ctx,\n+\t\t\t\t struct strbuf *base)\n {\n \tint i;\n \n \tassert(base->len == 0);\n \n-\tfor (i = 0; i < revs->pending.nr; i++) {\n-\t\tstruct object_array_entry *pending = revs->pending.objects + i;\n+\tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n+\t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n@@ -262,62 +254,49 @@ static void traverse_trees_and_blobs(struct rev_info *revs,\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tshow_object(obj, name, show_data);\n+\t\t\tctx->show_object(obj, name, ctx->show_data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(revs, (struct tree *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(revs, (struct blob *)obj, show_object,\n-\t\t\t\t base, path, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n \t\t oid_to_hex(&obj->oid), name);\n \t}\n-\tobject_array_clear(&revs->pending);\n+\tobject_array_clear(&ctx->revs->pending);\n }\n \n-static void do_traverse(struct rev_info *revs,\n-\t\t\tshow_commit_fn show_commit,\n-\t\t\tshow_object_fn show_object,\n-\t\t\tvoid *show_data,\n-\t\t\tfilter_object_fn filter_fn,\n-\t\t\tvoid *filter_data)\n+static void do_traverse(struct traversal_context *ctx)\n {\n \tstruct commit *commit;\n \tstruct strbuf csp; /* callee's scratch pad */\n \tstrbuf_init(&csp, PATH_MAX);\n \n-\twhile ((commit = get_revision(revs)) != NULL) {\n+\twhile ((commit = get_revision(ctx->revs)) != NULL) {\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n \t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(revs, get_commit_tree(commit));\n-\t\tshow_commit(commit, show_data);\n+\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tctx->show_commit(commit, ctx->show_data);\n \n-\t\tif (revs->tree_blobs_in_commit_order)\n+\t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n \t\t\t * NEEDSWORK: Adding the tree and then flushing it here\n \t\t\t * needs a reallocation for each commit. Can we pass the\n \t\t\t * tree directory without allocation churn?\n \t\t\t */\n-\t\t\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t\t\t show_object, show_data,\n-\t\t\t\t\t\t filter_fn, filter_data);\n+\t\t\ttraverse_trees_and_blobs(ctx, &csp);\n \t}\n-\ttraverse_trees_and_blobs(revs, &csp,\n-\t\t\t\t show_object, show_data,\n-\t\t\t\t filter_fn, filter_data);\n+\ttraverse_trees_and_blobs(ctx, &csp);\n \tstrbuf_release(&csp);\n }\n \n@@ -326,7 +305,14 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t\t show_object_fn show_object,\n \t\t\t void *show_data)\n {\n-\tdo_traverse(revs, show_commit, show_object, show_data, NULL, NULL);\n+\tstruct traversal_context ctx;\n+\tctx.revs = revs;\n+\tctx.show_commit = show_commit;\n+\tctx.show_object = show_object;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\tctx.filter_data = NULL;\n+\tdo_traverse(&ctx);\n }\n \n void traverse_commit_list_filtered(\n@@ -337,14 +323,18 @@ void traverse_commit_list_filtered(\n \tvoid *show_data,\n \tstruct oidset *omitted)\n {\n-\tfilter_object_fn filter_fn = NULL;\n+\tstruct traversal_context ctx;\n \tfilter_free_fn filter_free_fn = NULL;\n-\tvoid *filter_data = NULL;\n-\n-\tfilter_data = list_objects_filter__init(omitted, filter_options,\n-\t\t\t\t\t\t&filter_fn, &filter_free_fn);\n-\tdo_traverse(revs, show_commit, show_object, show_data,\n-\t\t filter_fn, filter_data);\n-\tif (filter_data && filter_free_fn)\n-\t\tfilter_free_fn(filter_data);\n+\n+\tctx.revs = revs;\n+\tctx.show_object = show_object;\n+\tctx.show_commit = show_commit;\n+\tctx.show_data = show_data;\n+\tctx.filter_fn = NULL;\n+\n+\tctx.filter_data = list_objects_filter__init(omitted, filter_options,\n+\t\t\t\t\t\t &ctx.filter_fn, &filter_free_fn);\n+\tdo_traverse(&ctx);\n+\tif (ctx.filter_data && filter_free_fn)\n+\t\tfilter_free_fn(ctx.filter_data);\n }\n-- \n2.19.1.331.ge82ca0e54c-goog\n\n"},{"id":"360342","messageId":"42ad5623bab52b8207831eab4a4f27e47a318e36.1539373969.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1539373969.git.matvore@google.com","subject":"[PATCH v12 2/8] list-objects: refactor to process_tree_contents","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-12T20:01:43Z","receivedAt":"2018-10-12T20:02:21Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This will be used in a follow-up patch to reduce indentation needed when\ninvoking the logic conditionally. i.e. rather than:\n\nif (foo) {\n\twhile (...) {\n\t\t/* this is very indented */\n\t}\n}\n\nwe will have:\n\nif (foo)\n\tprocess_tree_contents(...);\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 68 ++++++++++++++++++++++++++++++--------------------\n 1 file changed, 41 insertions(+), 27 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 584518a3f..ccc529e5e 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -94,6 +94,46 @@ static void process_gitlink(struct traversal_context *ctx,\n \t/* Nothing to do */\n }\n \n+static void process_tree(struct traversal_context *ctx,\n+\t\t\t struct tree *tree,\n+\t\t\t struct strbuf *base,\n+\t\t\t const char *name);\n+\n+static void process_tree_contents(struct traversal_context *ctx,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *base)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tenum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ?\n+\t\tall_entries_interesting : entry_not_interesting;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tif (match != all_entries_interesting) {\n+\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n+\t\t\t\t\t\t &ctx->revs->diffopt.pathspec);\n+\t\t\tif (match == all_entries_not_interesting)\n+\t\t\t\tbreak;\n+\t\t\tif (match == entry_not_interesting)\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (S_ISDIR(entry.mode))\n+\t\t\tprocess_tree(ctx,\n+\t\t\t\t lookup_tree(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t\telse if (S_ISGITLINK(entry.mode))\n+\t\t\tprocess_gitlink(ctx, entry.oid->hash,\n+\t\t\t\t\tbase, entry.path);\n+\t\telse\n+\t\t\tprocess_blob(ctx,\n+\t\t\t\t lookup_blob(the_repository, entry.oid),\n+\t\t\t\t base, entry.path);\n+\t}\n+}\n+\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n@@ -101,10 +141,6 @@ static void process_tree(struct traversal_context *ctx,\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n-\tstruct tree_desc desc;\n-\tstruct name_entry entry;\n-\tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n-\t\tall_entries_interesting: entry_not_interesting;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \tint gently = revs->ignore_missing_links ||\n@@ -144,29 +180,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tinit_tree_desc(&desc, tree->buffer, tree->size);\n-\n-\twhile (tree_entry(&desc, &entry)) {\n-\t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, 0,\n-\t\t\t\t\t\t &revs->diffopt.pathspec);\n-\t\t\tif (match == all_entries_not_interesting)\n-\t\t\t\tbreak;\n-\t\t\tif (match == entry_not_interesting)\n-\t\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t\telse if (S_ISGITLINK(entry.mode))\n-\t\t\tprocess_gitlink(ctx, entry.oid->hash, base, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n-\t}\n+\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n-- \n2.19.1.331.ge82ca0e54c-goog\n\n"},{"id":"360343","messageId":"0d1285a0ec412846e5417ec524a866f6b2c39d82.1539373969.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1539373969.git.matvore@google.com","subject":"[PATCH v12 3/8] list-objects: always parse trees gently","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-12T20:01:44Z","receivedAt":"2018-10-12T20:02:30Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"If parsing fails when revs->ignore_missing_links and\nrevs->exclude_promisor_objects are both false, we print the OID anyway\nin the die(\"bad tree object...\") call, so any message printed by\nparse_tree_gently() is superfluous.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex ccc529e5e..f9b51db7a 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,8 +143,6 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n-\tint gently = revs->ignore_missing_links ||\n-\t\t revs->exclude_promisor_objects;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -152,7 +150,7 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, gently) < 0) {\n+\tif (parse_tree_gently(tree, 1) < 0) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n-- \n2.19.1.331.ge82ca0e54c-goog\n\n"},{"id":"360344","messageId":"59856878ed196240db1365013d1822032ea66b83.1539373969.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1539373969.git.matvore@google.com","subject":"[PATCH v12 4/8] rev-list: handle missing tree objects properly","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-12T20:01:45Z","receivedAt":"2018-10-12T20:02:38Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Previously, we assumed only blob objects could be missing. This patch\nmakes rev-list handle missing trees like missing blobs. The --missing=*\nand --exclude-promisor-objects flags now work for trees as they already\ndo for blobs. This is demonstrated in t6112.\n\nSigned-off-by: Matthew DeVore \n---\n builtin/rev-list.c | 11 ++++---\n list-objects.c | 11 +++++--\n revision.h | 15 +++++++++\n t/t0410-partial-clone.sh | 45 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 13 ++++++++\n t/t6112-rev-list-filters-objects.sh | 22 +++++++++++++\n 6 files changed, 110 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 5b07f3f4a..49d6deed7 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -6,6 +6,7 @@\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"object.h\"\n #include \"object-store.h\"\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n@@ -209,7 +210,8 @@ static inline void finish_object__ma(struct object *obj)\n \t */\n \tswitch (arg_missing_action) {\n \tcase MA_ERROR:\n-\t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n+\t\tdie(\"missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tcase MA_ALLOW_ANY:\n@@ -222,8 +224,8 @@ static inline void finish_object__ma(struct object *obj)\n \tcase MA_ALLOW_PROMISOR:\n \t\tif (is_promisor_object(&obj->oid))\n \t\t\treturn;\n-\t\tdie(\"unexpected missing blob object '%s'\",\n-\t\t oid_to_hex(&obj->oid));\n+\t\tdie(\"unexpected missing %s object '%s'\",\n+\t\t type_name(obj->type), oid_to_hex(&obj->oid));\n \t\treturn;\n \n \tdefault:\n@@ -235,7 +237,7 @@ static inline void finish_object__ma(struct object *obj)\n static int finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\tif (!has_object_file(&obj->oid)) {\n \t\tfinish_object__ma(obj);\n \t\treturn 1;\n \t}\n@@ -373,6 +375,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tinit_revisions(&revs, prefix);\n \trevs.abbrev = DEFAULT_ABBREV;\n \trevs.commit_format = CMIT_FMT_UNSPECIFIED;\n+\trevs.do_not_die_on_missing_tree = 1;\n \n \t/*\n \t * Scan the argument list before invoking setup_revisions(), so that we\ndiff --git a/list-objects.c b/list-objects.c\nindex f9b51db7a..243192af5 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -143,6 +143,7 @@ static void process_tree(struct traversal_context *ctx,\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tint failed_parse;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -150,7 +151,9 @@ static void process_tree(struct traversal_context *ctx,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, 1) < 0) {\n+\n+\tfailed_parse = parse_tree_gently(tree, 1);\n+\tif (failed_parse) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n \n@@ -163,7 +166,8 @@ static void process_tree(struct traversal_context *ctx,\n \t\t is_promisor_object(&obj->oid))\n \t\t\treturn;\n \n-\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n+\t\tif (!revs->do_not_die_on_missing_tree)\n+\t\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \n \tstrbuf_addstr(base, name);\n@@ -178,7 +182,8 @@ static void process_tree(struct traversal_context *ctx,\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n-\tprocess_tree_contents(ctx, tree, base);\n+\tif (!failed_parse)\n+\t\tprocess_tree_contents(ctx, tree, base);\n \n \tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\ndiff --git a/revision.h b/revision.h\nindex 007278cc1..5910613cb 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -126,6 +126,21 @@ struct rev_info {\n \t\t\tline_level_traverse:1,\n \t\t\ttree_blobs_in_commit_order:1,\n \n+\t\t\t/*\n+\t\t\t * Blobs are shown without regard for their existence.\n+\t\t\t * But not so for trees: unless exclude_promisor_objects\n+\t\t\t * is set and the tree in question is a promisor object;\n+\t\t\t * OR ignore_missing_links is set, the revision walker\n+\t\t\t * dies with a \"bad tree object HASH\" message when\n+\t\t\t * encountering a missing tree. For callers that can\n+\t\t\t * handle missing trees and want them to be filterable\n+\t\t\t * and showable, set this to true. The revision walker\n+\t\t\t * will filter and show such a missing tree as usual,\n+\t\t\t * but will not attempt to recurse into this tree\n+\t\t\t * object.\n+\t\t\t */\n+\t\t\tdo_not_die_on_missing_tree:1,\n+\n \t\t\t/* for internal use only */\n \t\t\texclude_promisor_objects:1;\n \ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 128130066..5bc5b4445 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -186,6 +186,51 @@ test_expect_success 'rev-list stops traversal at missing and promised commit' '\n \t! grep $FOO out\n '\n \n+test_expect_success 'missing tree objects with --missing=allow-promisor and --exclude-promisor-objects' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo foo &&\n+\ttest_commit -C repo bar &&\n+\ttest_commit -C repo baz &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse bar^{tree}) &&\n+\tpromise_and_delete $(git -C repo rev-parse foo^{tree}) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-promisor --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 3 commits, 3 blobs, and 1 tree\n+\ttest_line_count = 7 objs &&\n+\n+\t# Do the same for --exclude-promisor-objects, but with all trees gone.\n+\tpromise_and_delete $(git -C repo rev-parse baz^{tree}) &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 3 commits, no blobs or trees\n+\ttest_line_count = 3 objs\n+'\n+\n+test_expect_success 'missing non-root tree object and rev-list' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\tmkdir repo/dir &&\n+\techo foo >repo/dir/foo &&\n+\tgit -C repo add dir/foo &&\n+\tgit -C repo commit -m \"commit dir/foo\" &&\n+\n+\tpromise_and_delete $(git -C repo rev-parse HEAD:dir) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\n+\tgit -C repo rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err &&\n+\ttest_must_be_empty rev_list_err &&\n+\t# 1 commit and 1 tree\n+\ttest_line_count = 2 objs\n+'\n+\n test_expect_success 'rev-list stops traversal at missing and promised tree' '\n \trm -rf repo &&\n \ttest_create_repo repo &&\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 6710c8bc8..ba83f3829 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -59,6 +59,19 @@ test_expect_success 'verify normal and blob:none packfiles have same commits/tre\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'get an error for missing tree object' '\n+\tgit init r5 &&\n+\techo foo >r5/foo &&\n+\tgit -C r5 add foo &&\n+\tgit -C r5 commit -m \"foo\" &&\n+\tdel=$(git -C r5 rev-parse HEAD^{tree} | sed \"s|..|&/|\") &&\n+\trm r5/.git/objects/$del &&\n+\ttest_must_fail git -C r5 pack-objects --rev --stdout 2>bad_tree <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgrep \"bad tree object\" bad_tree\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex d4ff0b3be..efe5a2467 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -195,6 +195,28 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' '\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for trees' '\n+\tTREE=$(git -C r3 rev-parse HEAD:dir1) &&\n+\n+\t# Create a spare repo because we will be deleting objects from this one.\n+\tgit clone r3 r3.b &&\n+\n+\trm r3.b/.git/objects/$(echo $TREE | sed \"s|^..|&/|\") &&\n+\n+\tgit -C r3.b rev-list --quiet --missing=print --objects HEAD \\\n+\t\t>missing_objs 2>rev_list_err &&\n+\techo \"?$TREE\" >expected &&\n+\ttest_cmp expected missing_objs &&\n+\n+\t# do not complain when a missing tree cannot be parsed\n+\ttest_must_be_empty rev_list_err &&\n+\n+\tgit -C r3.b rev-list --missing=allow-any --objects HEAD \\\n+\t\t>objs 2>rev_list_err &&\n+\t! grep $TREE objs &&\n+\ttest_must_be_empty rev_list_err\n+'\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.19.1.331.ge82ca0e54c-goog\n\n"},{"id":"360345","messageId":"38f39b252e2c0bbb7a9b48eba67849c92bb1f24b.1539373969.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1539373969.git.matvore@google.com","subject":"[PATCH v12 5/8] revision: mark non-user-given objects instead","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-12T20:01:46Z","receivedAt":"2018-10-12T20:02:47Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Currently, list-objects.c incorrectly treats all root trees of commits\nas USER_GIVEN. Also, it would be easier to mark objects that are\nnon-user-given instead of user-given, since the places in the code\nwhere we access an object through a reference are more obvious than\nthe places where we access an object that was given by the user.\n\nResolve these two problems by introducing a flag NOT_USER_GIVEN that\nmarks blobs and trees that are non-user-given, replacing USER_GIVEN.\n(Only blobs and trees are marked because this mark is only used when\nfiltering objects, and filtering of other types of objects is not\nsupported yet.)\n\nThis fixes a bug in that git rev-list behaved differently from git\npack-objects. pack-objects would *not* filter objects given explicitly\non the command line and rev-list would filter. This was because the two\ncommands used a different function to add objects to the rev_info\nstruct. This seems to have been an oversight, and pack-objects has the\ncorrect behavior, so I added a test to make sure that rev-list now\nbehaves properly.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects.c | 31 +++++++++++++++++------------\n revision.c | 1 -\n revision.h | 11 ++++++++--\n t/t6112-rev-list-filters-objects.sh | 12 +++++++++++\n 4 files changed, 39 insertions(+), 16 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 243192af5..7a1a0929d 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -53,7 +53,7 @@ static void process_blob(struct traversal_context *ctx,\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BLOB, obj,\n \t\t\t\t path->buf, &path->buf[pathlen],\n \t\t\t\t ctx->filter_data);\n@@ -120,17 +120,19 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\tcontinue;\n \t\t}\n \n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(ctx,\n-\t\t\t\t lookup_tree(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\tstruct tree *t = lookup_tree(the_repository, entry.oid);\n+\t\t\tt->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_tree(ctx, t, base, entry.path);\n+\t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid->hash,\n \t\t\t\t\tbase, entry.path);\n-\t\telse\n-\t\t\tprocess_blob(ctx,\n-\t\t\t\t lookup_blob(the_repository, entry.oid),\n-\t\t\t\t base, entry.path);\n+\t\telse {\n+\t\t\tstruct blob *b = lookup_blob(the_repository, entry.oid);\n+\t\t\tb->object.flags |= NOT_USER_GIVEN;\n+\t\t\tprocess_blob(ctx, b, base, entry.path);\n+\t\t}\n \t}\n }\n \n@@ -171,7 +173,7 @@ static void process_tree(struct traversal_context *ctx,\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn)\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n \t\tr = ctx->filter_fn(LOFS_BEGIN_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -185,7 +187,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (!failed_parse)\n \t\tprocess_tree_contents(ctx, tree, base);\n \n-\tif (!(obj->flags & USER_GIVEN) && ctx->filter_fn) {\n+\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n \t\tr = ctx->filter_fn(LOFS_END_TREE, obj,\n \t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t ctx->filter_data);\n@@ -301,8 +303,11 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n-\t\tif (get_commit_tree(commit))\n-\t\t\tadd_pending_tree(ctx->revs, get_commit_tree(commit));\n+\t\tif (get_commit_tree(commit)) {\n+\t\t\tstruct tree *tree = get_commit_tree(commit);\n+\t\t\ttree->object.flags |= NOT_USER_GIVEN;\n+\t\t\tadd_pending_tree(ctx->revs, tree);\n+\t\t}\n \t\tctx->show_commit(commit, ctx->show_data);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\ndiff --git a/revision.c b/revision.c\nindex de4dce600..72d48a17f 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -175,7 +175,6 @@ static void add_pending_object_with_path(struct rev_info *revs,\n \t\tstrbuf_release(&buf);\n \t\treturn; /* do not add the commit itself */\n \t}\n-\tobj->flags |= USER_GIVEN;\n \tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n }\n \ndiff --git a/revision.h b/revision.h\nindex 5910613cb..83e164039 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -21,9 +21,16 @@\n #define SYMMETRIC_LEFT\t(1u<<8)\n #define PATCHSAME\t(1u<<9)\n #define BOTTOM\t\t(1u<<10)\n-#define USER_GIVEN\t(1u<<25) /* given directly by the user */\n+/*\n+ * Indicates object was reached by traversal. i.e. not given by user on\n+ * command-line or stdin.\n+ * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n+ * filtering trees and blobs, but it may be useful to support filtering commits\n+ * in the future.\n+ */\n+#define NOT_USER_GIVEN\t(1u<<25)\n #define TRACK_LINEAR\t(1u<<26)\n-#define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n+#define ALL_REV_FLAGS\t(((1u<<11)-1) | NOT_USER_GIVEN | TRACK_LINEAR)\n \n #define DECORATE_SHORT_REFS\t1\n #define DECORATE_FULL_REFS\t2\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex efe5a2467..110d4f74d 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -30,6 +30,18 @@ test_expect_success 'verify blob:none omits all 5 blobs' '\n \ttest_cmp observed expected\n '\n \n+test_expect_success 'specify blob explicitly prevents filtering' '\n+\tfile_3=$(git -C r1 ls-files -s file.3 |\n+\t\t awk -f print_2.awk) &&\n+\n+\tfile_4=$(git -C r1 ls-files -s file.4 |\n+\t\t awk -f print_2.awk) &&\n+\n+\tgit -C r1 rev-list --objects --filter=blob:none HEAD $file_3 >observed &&\n+\tgrep \"$file_3\" observed &&\n+\t! grep \"$file_4\" observed\n+'\n+\n test_expect_success 'verify emitted+omitted == all' '\n \tgit -C r1 rev-list HEAD --objects \\\n \t\t| awk -f print_1.awk \\\n-- \n2.19.1.331.ge82ca0e54c-goog\n\n"},{"id":"360346","messageId":"65e48449b387a427c94117a76791656a4bfdb977.1539373969.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1539373969.git.matvore@google.com","subject":"[PATCH v12 6/8] list-objects-filter: use BUG rather than die","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-12T20:01:47Z","receivedAt":"2018-10-12T20:02:56Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"In some cases in this file, BUG makes more sense than die. In such\ncases, a we get there from a coding error rather than a user error.\n\n'return' has been removed following some instances of BUG since BUG does\nnot return.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter.c | 11 ++++-------\n 1 file changed, 4 insertions(+), 7 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex a0ba78b20..5f8b1a002 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -44,8 +44,7 @@ static enum list_objects_filter_result filter_blobs_none(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -102,8 +101,7 @@ static enum list_objects_filter_result filter_blobs_limit(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -208,8 +206,7 @@ static enum list_objects_filter_result filter_sparse(\n \n \tswitch (filter_situation) {\n \tdefault:\n-\t\tdie(\"unknown filter_situation\");\n-\t\treturn LOFR_ZERO;\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n@@ -389,7 +386,7 @@ void *list_objects_filter__init(\n \tassert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n \n \tif (filter_options->choice >= LOFC__COUNT)\n-\t\tdie(\"invalid list-objects filter choice: %d\",\n+\t\tBUG(\"invalid list-objects filter choice: %d\",\n \t\t filter_options->choice);\n \n \tinit_fn = s_filters[filter_options->choice];\n-- \n2.19.1.331.ge82ca0e54c-goog\n\n"},{"id":"360347","messageId":"9d3b59e0ca6f6b06b01ef7b2702528c1f3401364.1539373969.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1539373969.git.matvore@google.com","subject":"[PATCH v12 7/8] list-objects-filter-options: do not over-strbuf_init","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-12T20:01:48Z","receivedAt":"2018-10-12T20:03:04Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"The function gently_parse_list_objects_filter is either called with\nerrbuf=STRBUF_INIT or errbuf=NULL, but that function calls strbuf_init\nwhen errbuf is not NULL. strbuf_init is only necessary if errbuf\ncontains garbage, and risks a memory leak if errbuf already has a\nnon-STRBUF_INIT state. It should be the caller's responsibility to make\nsure errbuf is not garbage, since garbage content is easily avoidable\nwith STRBUF_INIT.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex c0e2bd6a0..d259bdb2c 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -30,7 +30,6 @@ static int gently_parse_list_objects_filter(\n \n \tif (filter_options->choice) {\n \t\tif (errbuf) {\n-\t\t\tstrbuf_init(errbuf, 0);\n \t\t\tstrbuf_addstr(\n \t\t\t\terrbuf,\n \t\t\t\t_(\"multiple filter-specs cannot be combined\"));\n@@ -71,10 +70,9 @@ static int gently_parse_list_objects_filter(\n \t\treturn 0;\n \t}\n \n-\tif (errbuf) {\n-\t\tstrbuf_init(errbuf, 0);\n+\tif (errbuf)\n \t\tstrbuf_addf(errbuf, \"invalid filter-spec '%s'\", arg);\n-\t}\n+\n \tmemset(filter_options, 0, sizeof(*filter_options));\n \treturn 1;\n }\n-- \n2.19.1.331.ge82ca0e54c-goog\n\n"},{"id":"360348","messageId":"0f3f7c3e1f06b6d13efe7d3688d1b0e5d0b37563.1539373969.git.matvore@google.com","threadId":"49086","inReplyTo":"cover.1539373969.git.matvore@google.com","subject":"[PATCH v12 8/8] list-objects-filter: implement filter tree:0","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-12T20:01:49Z","receivedAt":"2018-10-12T20:03:14Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Teach list-objects the \"tree:0\" filter which allows for filtering\nout all tree and blob objects (unless other objects are explicitly\nspecified by the user). The purpose of this patch is to allow smaller\npartial clones.\n\nThe name of this filter - tree:0 - does not explicitly specify that\nit also filters out all blobs, but this should not cause much confusion\nbecause blobs are not at all useful without the trees that refer to\nthem.\n\nI also considered only:commits as a name, but this is inaccurate because\nit suggests that annotated tags are omitted, but actually they are\nincluded.\n\nThe name \"tree:0\" allows later filtering based on depth, i.e. \"tree:1\"\nwould filter out all but the root tree and blobs. In order to avoid\nconfusion between 0 and capital O, the documentation was worded in a\nsomewhat round-about way that also hints at this future improvement to\nthe feature.\n\nSigned-off-by: Matthew DeVore \n---\n Documentation/rev-list-options.txt | 5 +++\n list-objects-filter-options.c | 13 +++++++\n list-objects-filter-options.h | 1 +\n list-objects-filter.c | 49 ++++++++++++++++++++++++++\n t/t5317-pack-objects-filter-objects.sh | 28 +++++++++++++++\n t/t5616-partial-clone.sh | 42 ++++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 15 ++++++++\n 7 files changed, 153 insertions(+)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 7b273635d..5f1672913 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -731,6 +731,11 @@ the requested refs.\n +\n The form '--filter=sparse:path=' similarly uses a sparse-checkout\n specification contained in .\n++\n+The form '--filter=tree:' omits all blobs and trees whose depth\n+from the root tree is >= (minimum depth if an object is located\n+at multiple depths in the commits traversed). Currently, only =0\n+is supported, which omits all blobs and trees.\n \n --no-filter::\n \tTurn off any previous `--filter=` argument.\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex d259bdb2c..e8da2e858 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -49,6 +49,19 @@ static int gently_parse_list_objects_filter(\n \t\t\treturn 0;\n \t\t}\n \n+\t} else if (skip_prefix(arg, \"tree:\", &v0)) {\n+\t\tunsigned long depth;\n+\t\tif (!git_parse_ulong(v0, &depth) || depth != 0) {\n+\t\t\tif (errbuf) {\n+\t\t\t\tstrbuf_addstr(\n+\t\t\t\t\terrbuf,\n+\t\t\t\t\t_(\"only 'tree:0' is supported\"));\n+\t\t\t}\n+\t\t\treturn 1;\n+\t\t}\n+\t\tfilter_options->choice = LOFC_TREE_NONE;\n+\t\treturn 0;\n+\n \t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n \t\tstruct object_context oc;\n \t\tstruct object_id sparse_oid;\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 0000a61f8..af64e5c66 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -10,6 +10,7 @@ enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n+\tLOFC_TREE_NONE,\n \tLOFC_SPARSE_OID,\n \tLOFC_SPARSE_PATH,\n \tLOFC__COUNT /* must be last */\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 5f8b1a002..09b2b05d5 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -79,6 +79,54 @@ static void *filter_blobs_none__init(\n \treturn d;\n }\n \n+/*\n+ * A filter for list-objects to omit ALL trees and blobs from the traversal.\n+ * Can OPTIONALLY collect a list of the omitted OIDs.\n+ */\n+struct filter_trees_none_data {\n+\tstruct oidset *omits;\n+};\n+\n+static enum list_objects_filter_result filter_trees_none(\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_trees_none_data *filter_data = filter_data_;\n+\n+\tswitch (filter_situation) {\n+\tdefault:\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n+\n+\tcase LOFS_BEGIN_TREE:\n+\tcase LOFS_BLOB:\n+\t\tif (filter_data->omits)\n+\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n+\n+\tcase LOFS_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\treturn LOFR_ZERO;\n+\n+\t}\n+}\n+\n+static void* filter_trees_none__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options,\n+\tfilter_object_fn *filter_fn,\n+\tfilter_free_fn *filter_free_fn)\n+{\n+\tstruct filter_trees_none_data *d = xcalloc(1, sizeof(*d));\n+\td->omits = omitted;\n+\n+\t*filter_fn = filter_trees_none;\n+\t*filter_free_fn = free;\n+\treturn d;\n+}\n+\n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n@@ -371,6 +419,7 @@ static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n+\tfilter_trees_none__init,\n \tfilter_sparse_oid__init,\n \tfilter_sparse_path__init,\n };\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex ba83f3829..d9dccf4d4 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -72,6 +72,34 @@ test_expect_success 'get an error for missing tree object' '\n \tgrep \"bad tree object\" bad_tree\n '\n \n+test_expect_success 'setup for tests of tree:0' '\n+\tmkdir r1/subtree &&\n+\techo \"This is a file in a subtree\" >r1/subtree/file &&\n+\tgit -C r1 add subtree/file &&\n+\tgit -C r1 commit -m subtree\n+'\n+\n+test_expect_success 'verify tree:0 packfile has no blobs or trees' '\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\t! grep -E \"tree|blob\" objs\n+'\n+\n+test_expect_success 'grab tree directly when using tree:0' '\n+\t# We should get the tree specified directly but not its blobs or subtrees.\n+\tgit -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF &&\n+\tHEAD:\n+\tEOF\n+\tgit -C r1 index-pack ../commitsonly.pack &&\n+\tgit -C r1 verify-pack -v ../commitsonly.pack >objs &&\n+\tawk \"/tree|blob/{print \\$1}\" objs >trees_and_blobs &&\n+\tgit -C r1 rev-parse HEAD: >expected &&\n+\ttest_cmp expected trees_and_blobs\n+'\n+\n # Test blob:limit=[kmg] filter.\n # We boundary test around the size parameter. The filter is strictly less than\n # the value, so size 500 and 1000 should have the same results, but 1001 should\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex bbbe7537d..392caa08f 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -154,6 +154,48 @@ test_expect_success 'partial clone with transfer.fsckobjects=1 uses index-pack -\n \tgrep \"git index-pack.*--fsck-objects\" trace\n '\n \n+test_expect_success 'use fsck before and after manually fetching a missing subtree' '\n+\t# push new commit so server has a subtree\n+\tmkdir src/dir &&\n+\techo \"in dir\" >src/dir/file.txt &&\n+\tgit -C src add dir/file.txt &&\n+\tgit -C src commit -m \"file in dir\" &&\n+\tgit -C src push -u srv master &&\n+\tSUBTREE=$(git -C src rev-parse HEAD:dir) &&\n+\n+\trm -rf dst &&\n+\tgit clone --no-checkout --filter=tree:0 \"file://$(pwd)/srv.bare\" dst &&\n+\tgit -C dst fsck &&\n+\n+\t# Make sure we only have commits, and all trees and blobs are missing.\n+\tgit -C dst rev-list --missing=allow-any --objects master \\\n+\t\t>fetched_objects &&\n+\tawk -f print_1.awk fetched_objects |\n+\txargs -n1 git -C dst cat-file -t >fetched_types &&\n+\n+\tsort -u fetched_types >unique_types.observed &&\n+\techo commit >unique_types.expected &&\n+\ttest_cmp unique_types.expected unique_types.observed &&\n+\n+\t# Auto-fetch a tree with cat-file.\n+\tgit -C dst cat-file -p $SUBTREE >tree_contents &&\n+\tgrep file.txt tree_contents &&\n+\n+\t# fsck still works after an auto-fetch of a tree.\n+\tgit -C dst fsck &&\n+\n+\t# Auto-fetch all remaining trees and blobs with --missing=error\n+\tgit -C dst rev-list --missing=error --objects master >fetched_objects &&\n+\ttest_line_count = 70 fetched_objects &&\n+\n+\tawk -f print_1.awk fetched_objects |\n+\txargs -n1 git -C dst cat-file -t >fetched_types &&\n+\n+\tsort -u fetched_types >unique_types.observed &&\n+\ttest_write_lines blob commit tree >unique_types.expected &&\n+\ttest_cmp unique_types.expected unique_types.observed\n+'\n+\n test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' '\n \trm -rf src dst &&\n \tgit init src &&\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 110d4f74d..08e0c7db6 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -229,6 +229,21 @@ test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for tre\n \ttest_must_be_empty rev_list_err\n '\n \n+# Test tree:0 filter.\n+\n+test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n+\tgit -C r3 rev-list --quiet --objects --filter-print-omitted \\\n+\t\t--filter=tree:0 HEAD >revs &&\n+\n+\tawk -f print_1.awk revs |\n+\tsed s/~// |\n+\txargs -n1 git -C r3 cat-file -t >unsorted_filtered_types &&\n+\n+\tsort -u unsorted_filtered_types >filtered_types &&\n+\ttest_write_lines blob tree >expected &&\n+\ttest_cmp expected filtered_types\n+'\n+\n # Delete some loose objects and use rev-list, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.19.1.331.ge82ca0e54c-goog\n\n"},{"id":"360475","messageId":"xmqqd0sc6ilk.fsf@gitster-ct.c.googlers.com","threadId":"49086","inReplyTo":"cover.1539373969.git.matvore@google.com","subject":"Re: [PATCH v12 0/8] filter: support for excluding all trees and blobs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-15T02:37:11Z","receivedAt":"2018-10-15T02:37:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthew DeVore writes:\n\n> Here is a re-roll-up since I haven't received any additional corrections for\n> almost a week.\n\nSorry, but doesn't this topic already in 'next'? If so, please make\nthese small clean-ups as incremental patches.\n\nThansk.\n"},{"id":"360478","messageId":"xmqqr2gr6fjy.fsf@gitster-ct.c.googlers.com","threadId":"49086","inReplyTo":"xmqqd0sc6ilk.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v12 0/8] filter: support for excluding all trees and blobs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-15T03:42:57Z","receivedAt":"2018-10-15T03:43:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Matthew DeVore writes:\n>\n>> Here is a re-roll-up since I haven't received any additional corrections for\n>> almost a week.\n>\n> Sorry, but doesn't this topic already sit in 'next'? If so, please make\n> these small clean-ups as incremental patches.\n\nHere is what I'd queue for now, with forged s-o-by from you ;-).\n\nThanks.\n\n-- >8 --\nFrom: Matthew DeVore \nDate: Fri, 12 Oct 2018 13:01:41 -0700\nSubject: [PATCH] filter-trees: code clean-up of tests\n\nA few trivial updates to test to match the current best practices.\n\n - avoid \"grep -q\" that strips potentially useful output from running\n tests under \"-v\".\n\n - use test_write_lines to prepare multi-line expected output file\n\n - reserve use of test_must_fail to \"git\" commands.\n\nSigned-off-by: Matthew DeVore \nSigned-off-by: Junio C Hamano \n---\n t/t5317-pack-objects-filter-objects.sh | 2 +-\n t/t5616-partial-clone.sh | 2 +-\n t/t6112-rev-list-filters-objects.sh | 6 +++---\n 3 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex 510d3537f6..d9dccf4d4d 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -69,7 +69,7 @@ test_expect_success 'get an error for missing tree object' '\n \ttest_must_fail git -C r5 pack-objects --rev --stdout 2>bad_tree <<-EOF &&\n \tHEAD\n \tEOF\n-\tgrep -q \"bad tree object\" bad_tree\n+\tgrep \"bad tree object\" bad_tree\n '\n \n test_expect_success 'setup for tests of tree:0' '\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 53fbf7db88..392caa08fd 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -192,7 +192,7 @@ test_expect_success 'use fsck before and after manually fetching a missing subtr\n \txargs -n1 git -C dst cat-file -t >fetched_types &&\n \n \tsort -u fetched_types >unique_types.observed &&\n-\tprintf \"blob\\ncommit\\ntree\\n\" >unique_types.expected &&\n+\ttest_write_lines blob commit tree >unique_types.expected &&\n \ttest_cmp unique_types.expected unique_types.observed\n '\n \ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 2cbb81d3bb..d24f9d5b5a 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -38,8 +38,8 @@ test_expect_success 'specify blob explicitly prevents filtering' '\n \t\t awk -f print_2.awk) &&\n \n \tgit -C r1 rev-list --objects --filter=blob:none HEAD $file_3 >observed &&\n-\tgrep -q \"$file_3\" observed &&\n-\ttest_must_fail grep -q \"$file_4\" observed\n+\tgrep \"$file_3\" observed &&\n+\t! grep \"$file_4\" observed\n '\n \n test_expect_success 'verify emitted+omitted == all' '\n@@ -241,7 +241,7 @@ test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n \txargs -n1 git -C r3 cat-file -t >unsorted_filtered_types &&\n \n \tsort -u unsorted_filtered_types >filtered_types &&\n-\tprintf \"blob\\ntree\\n\" >expected &&\n+\ttest_write_lines blob tree >expected &&\n \ttest_cmp expected filtered_types\n '\n \n-- \n2.19.1-328-g5a0cc8aca7\n\n"},{"id":"360628","messageId":"CAMfpvhK4a15gd-PT3W+4YJmpe6c7HyhJE5N_UqOzu8gsYYej4A@mail.gmail.com","threadId":"49086","inReplyTo":"xmqqr2gr6fjy.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v12 0/8] filter: support for excluding all trees and blobs","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2018-10-16T15:00:21Z","receivedAt":"2018-10-16T15:00:37Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"On Sun, Oct 14, 2018 at 8:43 PM Junio C Hamano wrote:\n>\n> Junio C Hamano writes:\n>\n> > Matthew DeVore writes:\n> >\n> >> Here is a re-roll-up since I haven't received any additional corrections for\n> >> almost a week.\n> >\n> > Sorry, but doesn't this topic already sit in 'next'? If so, please make\n> > these small clean-ups as incremental patches.\n>\n> Here is what I'd queue for now, with forged s-o-by from you ;-).\n>\n\nYes, this is fine, thank you! I've reapplied the patch in my own repo\non top of \"next\" in case I need to fix it and re-send, but please\nqueue what you have as-is.\n"}]}